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ABSTRACT 


Accurate  forecasts  of  U.S.  Navy  enlisted  end-strength  are  crucial  for 
budgetary  planning  and  the  development  of  manpower  policies.  An  improving 
economy  and  increased  employment  opportunities  in  the  civilian  sector  could 
cause  a  significant  problem  for  enlisted  retention.  The  Navy  Total  Force  Strength 
Model  (NTFSM)  is  a  new  stochastic  simulation  that  is  intended  to  offer  manpower 
analysts  more  accurate  enlisted  manpower  projections  than  those  projected  with 
the  current  tool.  NTFSM  uses  historical  data  and  user-defined  inputs  for 
economic  factors  to  project  monthly  retention  losses.  However,  NTFSM  is  still  in 
the  testing  phase  and  its  overall  behavior  is  largely  unknown.  In  particular,  the 
analysts  that  NTFSM  was  designed  to  help  are  unsure  of  the  effects  that  the 
economic  factors,  which  they  need  to  enter  themselves,  have  on  NTFSM’s 
output.  This  thesis  investigates  the  behavior  of  NTFSM’s  output  and  the 
sensitivity  of  the  user-entered  economic  factors.  Using  design  of  experiments 
and  data  mining,  a  variety  of  scenarios  are  simulated  and  then  analyzed  to  better 
understand  the  behavior  of  the  model  and  to  determine  the  sensitivity  of  the  user- 
defined  economic  factors.  The  results  of  the  analysis  unexpectedly  show  that 
NTFSM’s  economic  factors  have  no  significant  impact  on  NTFSM’s  end-strength 
output;  this  warrants  further  investigation. 
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EXECUTIVE  SUMMARY 


The  Navy’s  current  manpower  and  personnel  forecasting  tool  was 
developed  nearly  20  years  ago  and,  with  the  changing  budget  and  retention 
environment,  can  no  longer  keep  up  with  the  demands  of  modern  Navy 
manpower  and  personnel  analysis.  The  Navy  Total  Force  Strength  Model 
(NTFSM)  is  poised  to  replace  the  current  tool;  however,  it  is  still  in  its  testing 
phase.  NTFSM  is  an  agent-based,  stochastic  simulation  that  incorporates 
historical  data  and  user-defined  economic  factors  to  project  enlisted  personnel 
losses  and  gains  into  the  future.  NTFSM  has  undergone  initial  verification  testing, 
but  much  is  unknown  about  the  model  and  how  it  behaves.  This  study  serves  as 
an  initial  exploration  and  analysis  of  the  behavior  of  NTFSM  scenarios  under 
differing  economic  environments  and  also  as  a  proof  of  concept  for  simulation 
analysis  and  meta-modeling  techniques.  The  results  demonstrate  the  sensitivity 
of  NTFSM  outputs  to  changes  in  the  user-defined  economic  factors.  This 
information  can  be  used  to  help  manpower  and  personnel  analysts  better 
understand  NTFSM’s  strengths  and  weaknesses — and  eventually  lead  to  better 
utilization  of  NTFSM’s  capabilities. 

NTFSM  currently  resides  on  a  Navy  Manpower  Program  and  Budget 
System  (NMPBS)  testing  server  and  can  be  accessed  through  the  Navy  Total 
Force  Strength  Management  System  Graphical  User  Interface  (GUI).  The  GUI 
allows  for  the  user  to  create  unique  NTFSM  scenarios  and  access  NTFSM  output 
reports  which  can  then  be  analyzed. 

With  support  from  the  Simulation  Experiments  and  Efficient  Designs 
(SEED)  Center,  an  efficient  Nearly  Orthogonal  Latin  Hypercube  (NOLH)  Design 
of  Experiments  (DOE)  is  utilized  to  generate  NTFSM  output  data  that  covers  a 
wide  spectrum  of  economic  possibilities.  The  NOLH  design  used  varies 
economic  factors  to  efficiently  achieve  maximum  coverage  of  the  range  of 
possible  values.  The  experiment  in  this  study  utilizes  fiscal  year  2014  data  to 

project  one  fiscal  year  into  the  future  (FY2015).  However,  due  to  limited 

xvii 


computing  resources  available  on  the  NMPBS  testing  server  on  which  NTFSM  is 
currently  housed,  of  NTFSM’s  12  economic  factors,  only  seven  were  explored  in 
this  thesis.  In  addition,  of  the  numerous  outputs  NTFSM  produces,  only 
sensitivities  of  the  main  outputs  which  pertain  to  End  Strength  were  explored. 

Analysis  and  meta-modeling  of  the  data  generated  by  the  DOE  show  that, 
when  using  FY2014  data  to  project  one  year  into  the  future  (FY2015),  at  least 
some  stochastic  variation  is  present  in  all  of  NTFSM’s  main  outputs  and  most  are 
approximately  normally  distributed.  The  End  Strength  output  is  the  only  NTFSM 
output  which  is  not  normally  distributed  and  does  not  conform  to  any  of  the 
common  statistical  distributions.  The  End  Strength  output  also  experiences 
nearly  no  stochastic  variability,  with  an  estimated  mean  of  approximately  265,777 
and  a  standard  deviation  of  only  3.85;  this  result  warrants  further  investigation. 
Table  1  shows  which  of  NTFSM’s  economic  factors  have  an  effect  on  the  main 
NTFSM  outputs  explored. 


Table  1 .  Summary  of  NTFSM  Economic  Coefficients  that  Have 
an  Effect  on  the  NTFSM  Outputs  Explored 


Reenlistment 

Recruit  Losses 

Attrition  Losses 

Long  Extensions 

Unemployment 

Pay 

Unemployment 

Pay 

Unemployment 

Pay 

Unemployment 

Attrition  Losses 

NO 

NO 

NO 

NO 

YES 

YES 

NO 

Retirement  Losses 

NO 

NO 

NO 

NO 

YES 

NO 

NO 

Recruit  Losses 

YES 

NO 

NO 

NO 

YES 

NO 

YES 

EAOS  Losses 

YES 

NO 

NO 

NO 

NO 

NO 

YES 

Prior  Service  Gains 

NO 

NO 

NO 

YES 

YES 

YES 

YES 

Recruit  Gains 

YES 

NO 

NO 

NO 

YES 

NO 

NO 

End  Strength 

NO 

YES 

NO 

YES 

YES 

NO 

YES 

This  thesis  was  constrained  in  scope  by  the  available  computing 
resources  on  the  NMPBS  testing  server  on  which  NTFSM  is  currently  housed.  To 
improve  upon  this  limitation,  this  author  has  been  working  with  the  SEED  Center 
to  develop  a  method  for  transferring  the  NTFSM  simulation  and  historical  data 
repository  to  the  SEED  Center  high-speed  cluster  computing  server.  High-speed 
cluster  computing  opens  up  the  possibility  for  further  exploration  of  all  12  of 
NTFSM’s  economic  factors  (as  well  as  many  additional  factors),  thus  enabling 
the  ability  to  produce  more  generalized  results. 
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I.  INTRODUCTION 


The  Navy  had  an  estimated  end-strength  of  approximately  323,600 
(53,400  officers  and  270,200  enlisted  personnel)  in  fiscal  year  2014,  at  a  cost  of 
approximately  $45.4  billion  out  of  a  total  budget  of  $155.8  billion  (Department  of 
the  Navy,  2013).  Navy  enlisted  personnel  planning  is  a  difficult  and  complex 
process.  To  help  advise  senior  leadership  on  personnel  matters,  manpower 
analysts  at  the  Office  of  the  Chief  of  Naval  Operations  (OPNAV)  Manpower, 
Personnel  Training,  &  Education  (MPT&E)  Resource  Management  Division, 
Strategic  Resourcing  Branch  (N100)  use  a  deterministic  forecasting  tool  called 
the  Navy  Enlisted  Strength  Planning  (NESP)  model.  NESP  takes  the  population 
of  Enlisted  Sailors  who  are  eligible  to  leave  the  Navy  in  a  given  year  and  applies 
percentages  based  on  historical  data  to  categorize  and  quantify  expected 
retention  losses.  Realistically,  however,  the  model  is  not  used  for  forecasting,  as 
loss  percentages  are  calculated  outside  of  the  model  and  the  model  is  only  used 
to  help  determine  the  number  of  losses  by  paygrade  and  to  output  the  results  in 
the  format  that  the  Manpower  Budgeting  Office  (PERS-7)  requires. 

In  hopes  of  stemming  the  growing  proportion  of  resources  consumed  by 
personnel  costs,  the  Department  of  the  Navy  (DON)  has  put  an  emphasis  on 
efficient  manpower  management  by  implementing  new  talent  management 
initiatives  (Department  of  the  Navy,  2015).  NESP  does  not  have  the  ability  to  give 
much  insight  into  how  future  policy  changes  or  economic  factors  might  affect 
retention,  losses,  and  gains;  this  leaves  the  manpower  analysts  at  N100  unable 
to  efficiently  quantify  many  of  the  effects  that  any  economic  or  policy  changes 
may  have  on  future  personnel  budget  demands. 

The  margin  of  error  that  N100  has  to  operate  in  is  very  small.  The  required 
accuracy  of  manpower  forecasts  is  set  by  congress.  By  law,  the  Navy’s  total 
number  of  active  duty  personnel  at  the  end  of  the  fiscal  year  must  conform  to  the 
end-strength  guidelines  set  by  Congress  to  within  three  percent  above  or  .5 

percent  below  authorized  end-strength  (Title  10  United  States  Code).  The  Navy 
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realizes  that  a  more  robust  manpower  and  personnel  planning  tool  is  needed  in 
order  to  gain  the  insight  required  to  better  manage  the  enlisted  force  and  has 
begun  developing  a  new  manpower  personnel  model  called  the  Navy  Total  Force 
Strength  Management  (NTFSM)  model.  NTFSM  is  an  agent-based  stochastic 
manpower  simulation  that  uses  real  historical  data  pulled  from  Navy  Manpower 
Program  and  Budget  System  (NMPBS)  databases.  In  addition  to  using  historical 
data,  NTFSM  allows  its  users  to  input  economic  factors  and  policy  effects. 
Although  NTFSM  has  made  it  through  its  first  round  of  the  validation  process,  it  is 
still  in  the  testing  phase  and  very  little  is  known  about  the  model’s  behavior  or  the 
sensitivity  of  its  output  to  changes  in  the  user-defined  economic  factors  (S.  Cylke, 
personal  communication,  March  25,  2015). 

A.  PROBLEM  STATEMENT  AND  PURPOSE 

Before  NTFSM  is  ready  to  replace  NESP  as  N  100’s  main  manpower  and 
personnel  forecasting  tool,  more  needs  to  be  known  about  the  behavior  of 
NTFSM’s  output.  NTSFM  relies  on  a  “seeded”  random  number  generator  and, 
like  most  stochastic  simulations  that  do  so,  its  results  should  be  repeatable  if  the 
same  seed  is  used.  There  has  been  no  research,  however,  into  NTFSM’s  ability 
to  produce  repeatable  results  so  it  is  uncertain  if  NTFSM  possesses  this 
capability.  This  is  a  big  concern  for  N100  since  the  manpower  and  personnel 
forecasts  and  analyses  that  they  generate  are  used  by  top-level  decision  makers 
when  considering  changes  to  Navy-wide  manpower  and  personnel  policies,  and 
therefore  must  be  able  to  stand  up  to  extreme  scrutiny.  It  is  extremely  important 
to  ensure  that  NTFSM’s  output  is  repeatable  before  NTFSM  can  leave  the  testing 
phase. 

Additionally,  no  research  has  been  done  on  the  sensitivity  of  NTFSM’s 
main  simulation  outputs  to  changes  in  its  user-defined  economic  inputs.  As  it 
stands  now,  it  is  unknown  which  economic  inputs  affect,  and  to  what  degree  they 
affect,  the  model’s  outputs.  Since  NTFSM  is  a  stochastic  simulation  it  is 
important  to  run  several  iterations  of  the  simulation  in  order  to  gain  insight  on  the 
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distribution  of  possible  results.  A  large  number  of  runs  for  a  single  simulation 
scenario  has  never  been  attempted  on  NTFSM  and  therefore  the  variability  of  the 
model’s  output  is  largely  unknown. 

This  thesis  uses  a  quantitative  approach  to  better  understand  the 
strengths  and  weaknesses  associated  with  using  NTFSM  for  manpower  and 
personnel  forecasting.  This  is  done  using  the  current  graphical  user  interface 
(GUI)  as  it  appears  on  the  NMPBS  test  server.  A  robust  and  efficient 
experimental  design  is  developed  to  help  better  understand  the  limits  of  the 
model.  The  results  of  the  design  are  analyzed  using  advanced  statistical  and 
simulation  analysis  techniques  to  help  indicate  which  of  the  user-defined 
economic  factors  tested  have  the  greatest  impact  on  the  model’s  main  outputs 
and  to  gain  a  better  understanding  of  the  these  outputs’  behaviors. 

B.  RESEARCH  QUESTIONS,  METHODOLOGY,  AND  BENEFITS 

This  research  conducts  a  broad  exploratory  analysis  into  the  behavior  of 
NTFSM  under  a  variety  of  economic  scenarios,  by  using  an  efficient  experimental 
design,  to  gain  valuable  insight  into  the  model’s  general  behavior  and  to  help 
better  quantify  its  strengths  and  limitations.  The  following  questions  guide  the 
experimental  design  and  the  analysis  of  the  collected  data. 

1 .  Are  the  results  generated  by  NTFSM  repeatable? 

2.  What  is  the  general  behavior  of  NTFSM’s  main  outputs? 

3.  How  sensitive  are  NTFSM’s  main  outputs  to  changes  in  its  user- 
defined  economic  factors? 

This  research  utilizes  advanced  design  of  experiment  (DOE)  techniques 
developed  by  the  Simulation  Experiments  &  Efficient  Designs  (SEED)  Center,  an 
organization  within  the  Naval  Postgraduate  School  that  promotes  research  and 
advancement  of  simulation  analysis,  particularly  for  defense  applications.1  The 
user-defined  economic  factors  that  have  the  greatest  potential  for  volatility 
needed  to  be  initially  identified.  With  some  guidance  from  NIOO’s  manpower  and 

1  For  more  information,  visit  the  SEED  website  at  https://harvest.nps.edu. 
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personnel  analysts,  the  user-defined  economic  factors  chosen  to  be  varied  are 
those  pertaining  to  Expiration  of  Active  Obligated  Service  (EAOS)  losses,  Recruit 
gains,  Reenlistments,  Long  extensions,  and  Attritions.  All  other  factors  are  kept 
at  their  default  values.  Once  the  design  is  run  and  the  output  data  collected, 
advanced  statistical  methods  and  metamodeling  techniques  are  used  to  explore 
relationships  between  input  factors  and  model  output  to  develop  an 
understanding  of  the  response  surfaces  and  answer  the  research  questions. 

NTFSM  provides  a  promising  new  tool  that  can  be  used  to  gain  novel 
insights  into  manpower  and  personnel  forecasting.  These  insights  have  the 
potential  to  provide  valuable  information  to  top-level  Navy  decision  makers.  The 
insights  into  the  behavior  of  NTFSM  that  this  study  provides  helps  quantify  the 
behavior  of  the  model  and  gives  N  100’s  manpower  and  personnel  analysts  a 
better  overall  understanding  of  the  strengths  and  limitations  of  the  model,  which 
will,  in  turn,  allow  them  to  conduct  more  meaningful  analysis  of  the  model’s 
output. 

C.  LITERATURE  REVIEW 

This  study  focuses  on  the  use  of  design  of  experiments  to  gain  insight  into 
how  user-defined  economic  factors  affect  the  behavior  of  NTFSM’s  output.  Many 
studies  have  been  conducted  that  focus  on  the  effect  of  economic  factors  on 
manpower  and  retention,  such  as  in  Pinelis  and  Huff  (2014).  Designs  of 
experiments  have  been  used  in  academic  theses,  including  Erdman  (2010)  and 
DeHollan  (2015),  to  efficiently  explore  the  behavior  of  other  similar  manpower 
models,  such  as  the  Army’s  Enlisted  Specialty  model  and  the  Navy’s  Officer 
Strategic  Analysis  Model  (OSAM).  A  brief  review  of  some  of  these  studies 
follows. 


4 


1. 


Economic  Factors  and  Retention 


A  recent  Center  for  Naval  Analysis  study  focuses  on  the  relationship 
between  the  economy  and  the  decision  an  enlisted  Sailor  makes  on  reenlistment 
using  data  from  1992  to  2012  (Pinelis  &  Huff,  2014).  The  study  found  that 
economic  factors  could  affect  average  retention  percentages.  For  zone  A 
personnel  (personnel  with  less  than  six  years  of  active  service),  this  could  be  as 
much  as  a  25.1  percentage  point  increase  during  a  time  when  the  economy  is 
weak;  or,  a  decrease  in  average  retention  percentages  for  zone  A  personnel  by 
as  much  as  -21.8  percentage  points  during  a  time  when  the  economy  is  strong 
(Pinelis  &  Huff,  2014).  This,  of  course,  represents  only  the  most  extreme 
economic  scenarios,  but  it  can  serve  as  a  basis  for  the  constraining  bounds  of  an 
efficient  design  of  experiments. 

2.  Design  of  Experiments 

Erdman  (2010)  uses  design  of  experiments  to  explore  the  optimization 
component  of  the  U.S.  Army’s  Enlisted  Specialty  model,  which  is  an  enlisted 
manpower  model  that  is  used  to  minimize  the  deviation  between  Soldiers  on 
hand  and  authorized  positions  available  over  a  seven-year  planning  horizon.  The 
model  takes  into  account  859,633  variables  and  calculates  projections  against 
224,473  constraints.  Using  design  of  experiments,  Erdman  was  able  to  evaluate 
objective  function  coefficients  that  place  weights  on  decision  variables.  The 
results  of  the  study  led  to  an  average  drop  in  misaligned  Soldiers  of  8,355 
(equivalent  to  two  combat  brigades)  a  month  for  the  seven  year  planning  horizon 
(Erdman,  2010). 

DeHollan  (2015)  applies  design  of  experiments  and  data  farming 
techniques  to  OSAM  in  order  to  explore  the  effect  of  economic  factors  on  un¬ 
restricted  line  officer  end-strength.  Factors  with  the  greatest  potential  for  affecting 
end-strength  were  identified  and  varied  in  the  experiment.  The  resulting  output 
data  was  farmed  and  statistical  methods  were  used  to  explore  relationships 
between  factors.  Metamodeling  was  used  to  build  a  comprehensive 
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understanding  of  retention  issues.  DeHollan  (2015)  and  Erdman  (2010)  serve  as 
a  proof  of  concept  that  design  of  experiments  that  vary  economic  inputs  can  be 
used  to  gain  insight  into  the  output  and  overall  behavior  of  manpower  and 
personnel  models. 
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II.  NAVY  TOTAL  FORCE  STRENGTH  MANAGEMENT  SYSTEM 


This  chapter  gives  an  in  depth  overview  of  NTFSM,  including  additional 
information  on  NTFSM’s  development,  verification  and  testing,  and  design. 
NTFSM  is  an  agent-based  stochastic  model  which  utilizes  historical  manpower 
and  personnel  data  that  is  accessed  by  the  model  directly  through  the  Navy 
Manpower  Program  and  Budget  System  (NMPBS).  The  model  is  currently 
housed  on  an  NMPBS  testing  server,  but  it  is  intended  to  be  moved  to  a  NMPBS 
main  server  once  it  has  completed  its  testing  phase.  Users  access  NTFSM 
through  the  Navy  Total  Force  Strength  Management  System  Graphical  User 
Interface — where  model  scenarios  can  be  designed,  multiple  simulations  can  be 
run,  and  reports  can  be  generated.  This  study  accessed  NTFSM  solely  by 
utilizing  the  Navy  Total  Force  Strength  Management  System  Graphical  User 
Interface  and,  because  of  limited  computing  capacity  on  the  NMPBS  testing 
server,  this  study  was  somewhat  limited  on  the  scope  of  possible  simulation  runs. 

A.  NTFSM  DEVELOPMENT 

Total  force  strength  planning  and  execution  is  critical  to  OPNAV  N1 
Program  Objective  Memorandum  (POM)  analysis.  The  current  strength  model 
has  been  in  use  for  approximately  15  years  and  has  capability  shortfalls.  OPNAV 
N1  needs  a  more  timely  and  accurate  analysis  of  the  total  force  and  better 
connections  to  community-level  models  that  will  result  in  improvements  in 
operational  strength  and  readiness  (Department  of  the  Navy,  2011).  Because  of 
this,  the  Department  of  the  Navy  started  developing  a  new  manpower  and 
personnel  model.  The  Department  of  the  Navy  first  officially  began  development 
of  NTFSM  in  2011,  when  the  Navy  Total  Force  Strength  Model  Program  Plan 
gained  final  approval  from  N100,  N816M,  and  N14  (the  predecessor  of  the 
Strategic  Actions  Group).  The  purpose  of  the  Navy  Total  Force  Strength  Model 
Program  Plan  was  to  define  and  guide  project  efforts  to  develop  a  new  enlisted 
strength  model  that  would  assist  in  total  force  strength  planning,  analysis,  and 
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execution.  The  Navy  Total  Force  Strength  Model  Program  Plan  defined  the 
project’s  scope,  purpose,  objectives,  and  capabilities,  as  well  as  the  test, 
verification,  and  acceptance  terms  of  NTFSM  (Department  of  the  Navy,  2011).  In 
2012,  Serco,  a  DOD  contracting  agency  was  chosen  as  NTFSM’s  developer. 
Serco  developed  NTFSM  using  the  Navy  Total  Force  Strength  Model  Program 
Plan  as  guidance  for  the  model’s  capabilities.  A  list  of  the  required  model 
capabilities  as  mandated  by  the  Navy  Total  Force  Strength  Model  Program  Plan 
can  be  found  in  Appendix  A. 

B.  NTFSM  VERIFICATION  AND  TESTING 

The  Navy  Total  Force  Strength  Model  Program  Plan  mandates  that  a 
Testing  and  Verification  Plan  be  developed  for  NTFSM;  such  a  plan  was  drafted 
in  June  of  2013  and  executed  in  January  2015.  The  results,  published  in  Heider 
(2015),  tested  a  total  of  eight  requirement  categories: 

1 .  Navy  Total  Force  Strength  Model  Requirements 

2.  Personnel  Calculation  Requirements 

3.  End  Strength  Calculation  Requirements 

4.  Data  Repository  Requirements 

5.  Econometric  Calculation  Requirements 

6.  User  Interface  Requirements 

7.  Strength  Planning  Requirements 

8.  Analytical  Capability  Requirements 

The  Verification  and  Testing  Results  found  that  there  were  approximately 
56  total  requirements  mandated  by  the  Navy  Total  Force  Strength  Model 
Program  Plan  that  were  not  met,  and  approximately  72  that  were  only  partially 
met  (Heider,  2015).  Some  of  the  unmet  requirements  not  met  that  are  relevant  to 
this  study  include  the  requirement  to  verify  that  the  model  incorporates  (in  the 
personnel  calculations)  econometric  effects  to  Losses  by  Expiration  of  Active 
Obligated  Service,  Attrition,  and  Length  of  Service  (Heider,  2015).  This 
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requirement  was  tested  by  injecting  a  12  percent  unemployment  rate  into  a 
NTFSM  scenario.  It  was  found  that  there  was  no  statistical  difference  between 
this  scenario  and  a  baseline  scenario  that  used  the  default  “null  value”  setting  for 
the  unemployment  rate  factor.  A  second  simulation  scenario  was  run  which  set 
the  unemployment  rate  factor  to  12  percent,  and  also  set  all  other  economic 
factors  (referred  to  as  “Economic  Coefficients”  in  the  Navy  Total  Force  Strength 
Management  System  Graphical  User  Interface)  to  1.0.  It  was  found  that  the 
results  of  this  scenario  were  statistically  different  from  the  baseline  scenario.  The 
results  of  this  test  led  the  tester  to  conclude  that,  because  the  user  is  forced  to 
“guess”  the  extent  to  which  NTFSM’s  economic  coefficients  impact  the  variables 
by  manually  entering  values  for  these  coefficients,  the  adherence  to  the 
requirement  that  NTFSM  incorporates  economic  effects  to  losses  is  weak 
(Heider,  2015). 

This  author  was  not  able  to  find  any  other  information  on  the  verification 
and  testing  process  past  the  date  of  Heider  (2015),  but  speculates  that  N1  is 
working  with  the  developer  to  find  solutions  to  NTFSM’s  deviations  from  the 
requirements  mandated  by  the  Navy  Total  Force  Strength  Model  Program  Plan. 

C.  NTFSM  SOFTWARE  ARCHITECTURE  AND  ITS  WEAKNESSES 

The  Navy  Total  Force  Strength  Management  System  is  currently  housed 
on  an  NMPBS  test  server,  but  is  intended  to  be  moved  to  a  main  NMPBS  server 
once  it  has  completed  the  verification  and  testing  phase  and  is  officially  accepted 
by  the  Department  of  the  Navy.  The  Navy  Total  Force  Strength  Management 
System’s  Graphical  User  Interface  was  developed  using  Oracle  Application 
Express  (also  known  as  APEX).  The  historical  manpower  and  personnel  data 
repository  utilized  by  the  Navy  Total  Force  Strength  Management  System  is 
hosted  using  Oracle  database  software.  The  computational  implementation  of 
NTFSM’s  simulation  is  written  in  the  Java  programming  language.  The  Oracle 
APEX  and  data  repository  architecture  utilized  by  the  Navy  Total  Force  Strength 
Management  System  make  it  impossible  to  transfer  NTFSM  to  a  server  that  does 
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not  have  licensed  Oracle  and  APEX  software  installed.  If  NTFSM  were  able  to  be 
more  easily  transferred  to  other  servers  then  it  would,  in  theory,  be  possible  to 
utilize  High  Speed  Multi  Processor  Cluster  Computing  Systems,  such  as  the  one 
housed  at  the  Naval  Postgraduate  School’s  Simulation  Experiments  &  Efficient 
Designs  Center  for  Data  Farming.  Cluster  computing  could  potentially  allow  for 
the  analysis  of  the  entire  spectrum  of  NTFSM  simulation  output,  providing 
invaluable  insight  into  NTFSM’s  behavior,  and  giving  analysts  the  ability  to 
explore  NTFSM’s  true  potential  as  a  forecasting  tool. 

D.  NTFSM  GRAPHICAL  USER  INTERFACE  AND  SCENARIO  BUILDING 

The  Navy  Total  Force  Management  System  is  accessed  through  the 
NMPBS  Portal.  Users  must  first  request  an  account  to  allow  access  to  the 
NMPBS  server  before  being  able  to  log  in  via  Common  Access  Card  to  the  Navy 
Total  Force  Management  System  Graphical  User  Interface  Home  Page,  which  is 
shown  in  Figure  1 . 

Figure  1 .  Navy  Total  Force  Strength  Management  System  Graphical 

User  Interface  Home  Screen 


Source:  Navy  manpower  programming  and  budget  system,  (n.d.).  Accessed 
December  9,  2015.  https://nmpbst1.n10. npc.navy.mil/pls/nmpbstst/f?p=221:1: 
15989421618401 

The  Navy  Total  Force  Strength  Management  System  Graphical  User 
Interface  Home  Screen  serves  as  a  starting  point  for  accessing  NTFSM  and 
gives  up-to-date  information  on  the  latest  month  and  fiscal  year  of  historical  data 
available  in  the  Navy  Total  Force  Strength  Management  System’s  data 
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repository.  Although  the  repository  contains  historical  data  for  both  Active  Duty 
and  Full-Time  Support  Enlisted  Personnel,  this  study  focuses  on  the  Active  Duty 
component  of  the  data.  Currently,  the  data  repository  contains  historical 
manpower  and  personnel  data  for  Active  Duty  Navy  Enlisted  Personnel  starting 
from  October  of  fiscal  year  2005  to  February  of  fiscal  year  2015. 

Users  begin  the  scenario  building  process  by  accessing  the  Scenario 
Screen  via  the  Scenario  Tab  on  the  Navy  Total  Force  Strength  Management 
System  Graphical  User  Interface  Home  Screen.  A  snapshot  of  the  Navy  Total 
Force  Strength  Management  Systems  Graphical  User  Interface  Scenario  Screen 
is  shown  in  Figure  2. 


Figure  2.  Navy  Total  Force  Strength  Management  System  Graphical 

User  Interface  Scenario  Screen 


Source:  Navy  manpower  programming  and  budget  system,  (n.d.).  Accessed 
December  9,  2015.  https://nmpbst1.nl  0.npc.navy.mil/pls/nmpbstst/f?p=221:1: 
15989421618401 


From  the  Scenario  Screen  the  user  can  compare  the  policies  of  two 
previously  created  scenarios  of  their  choosing  by  using  the  “Compare”  button 
located  at  the  middle  right  side  of  the  page. 
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Clicking  the  “Create”  button  located  to  the  left  of  the  “Compare”  button 
opens  the  Scenario  Creation  Screen  where,  in  order  to  create  a  new  scenario, 
users  must  enter  a  scenario  name,  a  scenario  description,  choose  the  population 
group  (Active  or  Full  Time  Support),  choose  the  scenario  start  month  and  fiscal 
year,  and  choose  the  length  of  the  scenario  (from  one  to  10  fiscal  years).  Privacy 
settings  can  also  be  set  on  this  screen,  but,  after  reviewing  all  available  NTFSM 
documentation,  to  include  Serco  (2014),  and  after  speaking  to  Ms.  Elizabeth 
Heider,  who  performed  the  initial  verification  and  testing  on  NTFSM  and  authored 
the  NTFSM  Verification  and  Testing  Report,  this  author  has  not  been  able  to 
determine  how  the  privacy  settings  affect  the  scenario  since  the  Navy  Total 
Force  Strength  Management  System  Graphical  User  Interface  does  not  allow 
users  direct  access  to  any  Personally  Identifiable  Information  (Pll). 

After  creating  a  new  scenario  or  selecting  a  scenario  that  has  previously 
been  created  and  saved,  the  Scenario  Details  Screen  is  displayed.  The  Scenario 
Details  Screen  allows  for  easy  access  to  the  scenario’s  Policy  Screens  and  any 
report  sets  that  have  been  previously  generated.  There  are  a  total  of  nine  Policy 
screens,  one  each  for  policies  pertaining  to  Attrition,  Economics,  Prior  Service 
Gains,  Demotion,  Exam  Advancements,  Retirements,  Expiration  of  Active 
Obligated  Service,  Non-Prior  Service  gains,  and  Un-exam  Related 
Advancements.  Figure  3  shows  a  snapshot  of  the  Scenario  Details  Screen. 
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Figure  3.  Navy  Total  Force  Strength  Management  System  Graphical 
User  Interface  Scenario  Details  Screen 


Source:  Navy  manpower  programming  and  budget  system,  (n.d.).  Accessed 
December  9,  2015.  https://nmpbst1.nl 0.npc.navy.mil/pls/nmpbstst/f?p=221:1: 
15989421618401 


This  study  focuses  on  the  user-defined  economic  factors  that  can  be 
modified  on  the  Economic  Policy  Screen;  specifically,  those  that  are  referred  to 
on  the  Graphical  User  Interface  as  “Coefficients.”  The  model  separates  economic 
factors  into  two  categories:  “Conditions”  and  “Coefficients.”  There  are  four  total 
economic  conditions  that  the  user  can  modify:  unemployment,  inflation,  civilian 
wage  growth,  and  military  wage  growth.  Heider  (2015),  however,  reports  that 
changing  the  unemployment  condition  from  its  default  “null”  value  to  12  percent 
has  no  effect  on  NTFSM’s  output.  It  is  unknown  how,  or  if,  the  other  economic 
conditions  have  an  effect  on  NTFSM’s  output.  This  thesis  research  leaves  the 
analysis  of  NTFSM’s  economic  conditions  to  a  future  study  and  focuses  on  the 
analysis  of  NTFSM’s  economic  coefficients. 

NTFSM’s  economic  coefficients  are  comprised  of  six  events: 
Reenlistments,  Long  Extensions,  Prior  Service  Gains,  Recruit  Losses,  Attrition 
Losses,  and  Retirement  Losses.  Each  of  these  six  events  has  two  coefficients 
associated  with  them,  one  coefficient  for  the  National  Unemployment  Rate  and 
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the  other  for  the  Pay  Variable.  This  author  was  unable  to  find  the  definition  of 
“Pay  Variable”  in  any  of  the  NTFSM  documentation,  but  assumes  that  the  Pay 
Variable  refers  to  either  the  Military  Pay  Rate  or  the  difference  between  civilian 
and  military  pay  growth.  In  any  case,  according  to  Heider  (2015)  it  is  unknown 
how,  or  to  what  extent,  the  unemployment  and  pay  variable  coefficients  effect 
NTFSM’s  output.  What  is  known  about  the  economic  coefficients  is  that  the 
amounts  entered  into  the  Graphical  User  Interface  are  converted  to  percentages 
before  being  used  by  the  simulation  (Serco,  2014).  A  snapshot  of  the  Economic 
Policy  Screen  with  example  economic  coefficients  entered  is  shown  in  Figure  4. 


Figure  4.  Navy  Total  Force  Strength  Management  System  Graphical 
User  Interface  Economic  Policy  Screen 


Source:  Navy  manpower  programming  and  budget  system,  (n.d.).  Accessed 
December  9,  2015.  https://nmpbst1.nl 0.npc.navy.mil/pls/nmpbstst/f?p=221:1: 
15989421618401 


E.  NTFSM  REPORTS 

For  each  scenario  that  is  simulated  by  NTFSM,  there  are  approximately 
eight  reports  generated.  They  are  labeled  Advancement,  BLUF,  Costs,  LOS, 

Monthly  Summary,  PG  Summary,  Uncertainty-Grades,  and  Uncertainty-Years. 
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Raw  model  output  data  is  also  generated  and  is  saved  in  the  Comma  Separated 
Value  format  type.  The  data  are  separated  into  four  categories  labeled  End 
Strength,  End  Strength  by  FY,  Simulation  Events,  and  Event  &  End  Strength 
Comparison.  These  reports  and  data  can  be  accessed  via  the  Navy  Total  Force 
Strength  Management  System  Graphical  User  Interface’s  Reports  Screen,  or 
Scenario  Details  Screen.  However,  neither  this  author  nor  the  SEED  Center 
technical  staff  has  been  able  to  successfully  download  any  of  the  data  files 
generated  by  NTFSM.  A  snapshot  of  the  Navy  Total  Force  Strength  Management 
System  Graphical  User  Interface  Reports  Selection  Window  is  shown  in  Figure  5. 

Figure  5.  Navy  Total  Force  Strength  Management  System  Graphical 
User  Interface  Reports  Selection  Window 


View  Reports 

Simulation  Reports 

•  Advancement 

•  BLUF 

-  Costs 

•  LOS 

•  Monthly  Summary 

•  PG  Summary 

•  Uncertainty  -  Grades 

-  Uncertainty  -  Years 

Simulation  Data 

-  End  Strength 

•  End  Strenath  bv  FY 

•  Simulation  Events 

Cancel 

Source:  Navy  manpower  programming  and  budget  system,  (n.d.).  Accessed 
December  9,  2015.  https://nmpbst1.n10. npc.navy.mil/pls/nmpbstst/f?p=221:1: 
15989421618401: 

This  study  focuses  on  the  Monthly  Summary  and  Uncertainty-Years 

Reports.  The  Monthly  Summary  Report  is  generated  in  the  format  required  by  the 

Manpower  Budgeting  Office.  The  Monthly  Summary  Report  includes:  Losses  by 

month,  broken  down  by  Prior  Service  Gains  and  Non-Prior  Service  Gains  (new 

recruits);  Losses  by  month,  broken  down  by  Attrition  losses;  Expiration  of  Active 
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Obligated  Service  (EAOS)  Loses;  Retirement  Losses;  and  Trainee  Losses.  An 
example  Monthly  Summary  Report  is  shown  in  Figure  6. 


Figure  6.  Navy  Total  Force  Strength  Model  Monthly  Summary  Report 


Start:  FY 1 5  -  OCT  Number  Of  FYs:  1  Population  Group:  Active  #  Executions:  30 

For  sims  that  do  not  start  in  Oct.  Report  will  use  history  for  missing  months  data 

Strength  Prev  Sep 

Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total  with  OCs  266,703 

266,584 

266,466 

266,346 

266,225 

266,110 

265,990 

265,871 

265,754 

265,634 

265,516 

265,396 

265,278 

Enlisted  OCs  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Total  without  OCs  266,703 

266,584 

266  466 

260,346 

266.225 

206.110 

265,000 

265.871 

205.754 

265.634 

265.510 

205.306 

205,278 

WYA  by  Month  0 

266.644 

206.525 

266,406 

260.280 

206,107 

266,050 

265.030 

265.812 

265.604 

205.575 

205,456 

265,337 

Gains 

Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

PSG  Gains 

50 

36 

37 

62 

55 

45 

45 

37 

23 

35 

27 

23 

486 

ADOS 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

Prior  Service 

33 

20 

12 

31 

25 

17 

14 

12 

6 

6 

4 

4 

185 

Other  Gains 

26 

16 

25 

32 

30 

28 

31 

25 

17 

20 

23 

10 

200 

NPS  Gains 

2,800 

2.525 

2.301 

2,002 

2.508 

2.228 

2,042 

3.250 

2,757 

3,500 

2,070 

2,078 

33,362 

TEAOS  0  to  3  Years 

0 

0 

3 

0 

0 

0 

2 

0 

0 

1 

2 

0 

7 

TEAOS  4  Years 

2,867 

2.401 

2.350 

2,880 

2.552 

2.220 

2.610 

3.212 

2,741 

3,572 

2.058 

2.004 

33,144 

TEAOS  5  Years 

14 

24 

0 

10 

8 

5 

11 

23 

14 

14 

0 

10 

143 

TEAOS  6+  Years 

8 

11 

0 

3 

8 

3 

11 

15 

1 

3 

2 

4 

60 

Total:  Gains 

2,949 

2,562 

2,399 

2,964 

2,623 

2,273 

2,687 

3,287 

2,780 

3,625 

2,998 

2,701 

33,848 

Losses 

Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

Attrition  Losses 

1,240 

1,005 

1.120 

1,300 

1.155 

1.102 

1,350 

1.801 

1,304 

1,602 

1.270 

1,281 

15,743 

Physical 

426 

400 

402 

688 

400 

307 

470 

473 

004 

814 

012 

570 

0,423 

Hardship 

20 

47 

24 

13 

30 

30 

38 

40 

10 

27 

22 

10 

340 

Cause 

601 

402 

402 

547 

470 

508 

645 

640 

452 

500 

423 

300 

0,250 

Miscellaneous 

103 

151 

151 

140 

180 

106 

101 

033 

220 

102 

222 

200 

2,724 

EAOS  Losses 

1,057 

702 

006 

1,000 

034 

006 

850 

884 

830 

1.270 

030 

700 

10,741 

Zone  A  :  LOS  <=  6  years 

738 

510 

444 

608 

508 

435 

573 

562 

568 

030 

020 

407 

7.180 

Zone  B  :  LOS  7  to  10  years 

228 

208 

182 

208 

263 

100 

223 

245 

202 

250 

240 

211 

2,655 

Zone  C  :  LOS  11  to  14  years 

74 

62 

57 

80 

60 

62 

40 

66 

57 

71 

52 

47 

737 

Zone  D  :  LOS  15  to  20  years 

16 

12 

13 

14 

12 

0 

14 

11 

11 

17 

12 

13 

154 

Zone  E  :  LOS  >=  21  years 

0 

0 

0 

0 

2 

1 

0 

0 

0 

1 

1 

1 

0 

Retirement  Losses 

411 

454 

401 

387 

360 

301 

342 

308 

477 

517 

515 

401 

5,000 

Trainee  Losses 

351 

338 

202 

301 

283 

234 

250 

320 

281 

340 

385 

308 

3.600 

Total:  Losses 

3,067 

2,680 

2,518 

3,085 

2,739 

2,393 

2,807 

3,404 

2,900 

3,743 

3,117 

2,819 

35,273 

Source:  Navy  manpower  programming  and  budget  system,  (n.d.).  Accessed 
December  9,  2015.  https://nmpbst1.n10. npc.navy.mil/pls/nmpbstst/f?p=221:1: 
15989421618401 


Notice  that  the  first  line  of  the  report  shown  in  Figure  6  lists  the  simulation 
start  month  and  fiscal  year,  the  number  of  fiscal  years  the  simulation  was  run  for, 
the  population  group,  and  the  number  of  times  the  simulation  was  executed.  The 
simulation  for  the  scenario  that  generated  the  Monthly  Summary  Report  shown  in 
Figure  6  was  executed  thirty  times.  In  other  words,  the  simulation  ran  through 
thirty  iterations  of  this  particular  scenario.  Notice  also  in  Figure  6  that  only  one 
number  is  reported  for  each  of  the  outputs  listed  in  the  Monthly  Summary  Report. 
The  numbers  shown  in  the  report  are  the  estimated  means,  which  were 
calculated  from  the  output  of  thirty  iterations  of  the  NTFSM  simulation.  NTFSM, 
however,  is  a  stochastic  model  and  therefore  means  by  themselves  provide 

16 


insufficient  insight  without  some  measure  of  variability,  such  as  a  standard 
deviation  or  standard  error. 

The  Uncertainty-Years  Report  includes  estimated  means  and  standard 
errors  for  Begin  Strength,  Reenlistments,  Short  and  Long  Extensions,  Expiration 
of  Active  Obligated  Service  (EAOS)  Losses,  Retirement  Losses,  OCS  Starts, 
OCS  Graduations,  OCS  Failures,  Attrition  Losses,  Prior  Service  Gains,  Recruit 
Gains,  Recruit  Losses,  Unexamined  Advancements,  Demotions,  Examined 
Advancements,  and  End  Strength,  broken  down  by  fiscal  year.  Since  the 
Uncertainty-Years  Report  offers  a  measure  of  the  variance  of  the  distribution  of 
NTFSM  outputs,  the  outputs  this  study’s  analysis  focuses  on  were  chosen  from 
this  report.  An  example  Uncertainty-Years  report  is  shown  in  Figure  7. 


Figure  7.  Navy  Total  Force  Strength  Model  Uncertainty-Years  Report 


Start:  FY1 5  -OCT  Number  Of  FYs:  1  Population  Group:  Active  #  Executions:  30 
For  sims  that  do  not  start  in  Oct.  Report  will  NOT  use  history  for  missing  months  data 


Uncertainty  Report  -  Years 


Begin 

Strength 

Reenlist¬ 

ment 

Extensions 

Short  Long 

EAOS 

Losses 

Retirement 

Losses 

OCS 

Starts  Grads 

Failures 

Attrition 

Losses 

PS 

Gains 

Recruit 

Gains 

Losses 

Unexamined 

Advancements 

Demotions 

Examined 

Advancements 

End 

Strength 

FY15  Estimate  266.703 

40.861 

17.271  6.529 

10.741 

5.000 

0  0 

0 

15.743 

488 

33.362 

3,699 

49.910 

5.518 

45.282 

265.278 

FY15  Std  Error  0 

134 

103  74 

94 

31 

o  <r 

0 

126 

22 

211 

71 

167 

84 

69 

4 

_ 

Estimated  mean  values  (first  line  of  data)  are  the  total  values  for  the  fiscal  year 
shown  in  the  rightmost  column.  Source:  Source:  Navy  manpower  programming 
and  budget  system.  (n.d.).  Accessed  December  9,  2015. 

https://nmpbst1.n10.npc.navy.mil/pls/nmpbstst/f?p=221:1:1 5989421 61 8401 


F.  SIMULATION  RUN  TIME  OVERVIEW 

The  run  time  for  a  single  iteration  of  a  NTFSM  scenario  depends  on  the 
user-defined  time  horizon  and  the  current  traffic  on  the  NMPBS  server  network.  A 
single  iteration  of  a  NTFSM  scenario  which  projects  over  a  time-horizon 
consisting  of  a  single  fiscal  year  typically  takes  anywhere  from  five  to  30  minutes. 
Once  the  simulation  has  completed,  a  report  set  is  generated  and  NTFSM  closes 
access  to  the  Policy  Screens  so  that  modification  of  the  NTFSM  scenario’s 
policies  cannot  occur.  Chapter  IV  of  this  thesis  contains  a  more  in  depth 
assessment  of  NTFSM’s  run  time. 
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III.  EXPERIMENT  DESIGN  AND  IMPLEMENTAION 


This  chapter  discusses  the  use  of  design  of  experiments  (DOE)  to 
generate  NTFSM  output  that  covers  a  broad  range  of  possible  scenarios  to  be 
analyzed.  In  order  to  develop  a  design  of  experiments  that  provides  insight  into 
the  overall  behavior  of  NTFSM,  an  efficient  design  that  allows  for  the  analysis  of 
many  possible  response  surfaces  is  selected. 

A.  DESIGN  OF  EXPERIMENTS 

Applying  design  of  experiments  to  simulations  enables  us  to  gain  insight 
into  the  underlying  system  of  processes  that  lead  to  the  generation  of  simulation 
output  values.  Simulation  output  values  provide  little  meaningful  information 
unless  the  proper  context  is  applied.  Designs  of  experiments  enable  us  to  better 
understand  the  system  in  which  those  output  values  arise  and  to  explore  the 
effect  of  potential  policy  changes  on  those  systems  (Kleijnen,  Sanchez,  Lucas,  & 
Cioppa,  2005).  Designs  of  experiments  allow  us  to  run  a  set  of  experimental 
scenarios  which  efficiently  sample  from  the  total  spectrum  of  possibilities.  The 
data  collected  from  a  design  of  experiment  can  then  be  used  to  develop  meta¬ 
models  that  can  provide  insight  on  which,  and  to  what  extent,  simulation  inputs 
affect  simulation  outputs. 

B.  DESIGN  SELECTION 

There  are  a  number  of  considerations  that  help  guide  in  the  selection  of  an 

appropriate  design  of  experiments.  A  number  of  choices  along  a  spectrum  of 

complexity  give  the  designer  flexibility  in  the  approach,  although  time  and 

computing  resources  remain  a  constraint  (Kleijnen  et  al.,  2005).  This  study  relies 

on  the  Navy  Total  Force  Strength  Management  System  for  the  implementation 

and  execution  of  a  design  of  experiments  that  has  the  ability  to  generate  a  good 

representation  of  the  response  space.  The  Navy  Total  Force  Management 

System  is  constrained  by  the  available  computing  capacity  of  the  NMPBS  testing 

server  on  which  it  is  housed;  therefore,  the  only  feasible  designs  for  this  study 
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are  those  which  require  a  relatively  small  amount  of  computing  resources  while 
still  allowing  for  the  analysis  of  many  diverse  response  surfaces.  Latin  hypercube 
designs  arise  as  a  good  candidate  for  this  study.  These  designs  are  well  suited  to 
studies  in  which  gaining  a  better  understanding  of  the  response  surface  is  a 
primary  goal,  as  they  enable  the  fitting  of  many  diverse  response  surfaces 
(Sanchez  &  Wan,  2012).  Further  efficiency  and  improved  space-filling  properties 
can  be  gained  by  using  a  nearly  orthogonal  Latin  hypercube  (NOLH)  design 
(Cioppa  &  Lucas,  2007) 

Traditionally,  a  complete  understanding  of  a  response  surface  could  be 
achieved  with  a  full  factorial  design  that  iterates  through  every  possible  factor 
value  combination.  Unfortunately,  this  requires  an  extremely  large  amount  of 
computing  resources  as  designs  grow  exponentially  as  factors  and  levels  are 
added  and  quickly  become  unmanageable.  For  example,  a  full  factorial  design 
that  explores  a  single  replication  of  only  seven  factors,  each  of  which  has  only  10 
possible  settings,  would  consist  of  282,475,249  design  points.  To  put  this  in 
perspective,  if  each  design  point  was  run  30  times  and  it  took  only  one  second  to 
process  a  run,  then  it  would  take  approximately  268  years  to  run  the  entire 
design.  Nearly  orthogonal  Latin  hypercube  designs  allow  for  a  thorough  analysis 
of  the  response  while  requiring  only  a  small  fraction  of  the  number  of  design 
points  of  a  traditional  full  factorial  design  due  to  their  space-filling  ability.  A  NOLH 
design  can  explore  seven  factors  while  only  requiring  17  design  points.  To  put 
this  in  perspective,  if  each  design  point  were  run  30  times  and  it  took  one  second 
to  execute  one  run,  then  it  would  only  take  about  8.5  minutes  to  run  the  entire 
design.  NOLH  designs  are  able  to  achieve  this  extreme  level  of  efficiency  by 
efficiently  scattering  design  points  throughout  the  design  space  in  a  way  that 
achieves  a  space-filling  pattern  that  is  able  to  capture  a  very  large  portion  of  the 
range  of  possibilities  while  requiring  a  very  small  number  of  design  points. 
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C.  FACTOR  SELECTION 

Simulation  analysis  and  design  of  experiments  terminology  refers  to  a 
‘factor’  as  a  parameter,  variable,  or  input  to  a  simulation.  The  choice  of  factors  for 
a  design  of  experiments  depends  on  the  intent  of  the  experiment,  the 
characteristics  of  the  available  factors,  and  the  computing  resources  available  to 
run  the  experiment  (Kleijnen  et  al.,  2005).  The  number  of  factors  chosen  for  this 
study  is  mainly  influenced  by  the  availability  of  computing  resources.  Although  it 
would  be  ideal  to  explore  all  12  of  NTFSM’s  Economic  Coefficients,  the  NMPBS 
testing  server,  on  which  NTFSM  is  currently  housed,  does  not  have  the 
computing  capacity  required  to  run  the  65  design  points  required  for  a  12-factor 
NOLH  design  using  the  SEED  Center’s  online  design  spreadsheet.  A  NOLFI 
design  that  explores  seven  factors,  however,  requires  only  17  design  points  and 
therefore  needs  approximately  a  quarter  of  the  time  and  computing  resources  of 
a  12  factor  (or  65  design  point)  NOLH  design.  Using  more  than  seven  factors, 
because  of  the  nature  of  NOLHs,  would  require  a  design  consisting  of  at  least  33 
design  points,  which  would  require  about  twice  the  processing  time  to  complete 
and  could  potentially  cause  a  strain  on  the  computing  resources  of  the  NMBPS 
testing  server.  Therefore,  a  seven-factor  NOLH  design  was  chosen  for  this  study. 
The  NTFSM  economic  coefficients  that  were  chosen  as  factors  for  the  design  are 
shown  in  Table  1.  These  economic  coefficients  are  those  that  pertain  to  the 
events  that  are  either  the  most  important,  or  contain  the  most  uncertainty  and 
highest  variability  and  therefore  are  the  most  difficult  to  accurately  project  using 
the  current  system. 
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Table  1 .  List  of  Factors  Used  to  Build  Design 


Event 

Unemployment 

Pay 

Reenlistments 

X 

X 

Long  Extensions 

X 

Prior  Service  Gains 

Recruit  Losses 

X 

X 

Attrition  Losses 

X 

X 

Retirement  Losses 

X’s  denote  Economic  Coefficients  used  as  factors  in  the  design. 


D.  FACTOR  RANGE  DETERMINATION 

In  order  to  produce  a  NOLH  design,  a  suitable  range  of  values  for  each  of 
the  chosen  factors  must  first  be  determined.  A  recent  Center  for  Naval  Analysis 
study  into  the  effects  of  economic  variables  and  Navy  enlisted  retention  titled  The 
Economy  and  Enlisted  Retention  in  the  Navy  (Pinelis  &  Huff,  2014)  found  that 
economic  variables,  including  the  national  unemployment  rate  and  the  real 
disposable  personal  income  growth  rate,  could  have  as  much  as  a  25.1  percent 
positive  effect  on  enlisted  retention  in  the  Navy  when  the  state  of  the  economy  is 
extremely  weak,  and  as  much  as  a  -20.9  percent  negative  effect  on  enlisted 
retention  in  the  Navy  when  the  state  of  the  economy  is  extremely  strong.  These 
results  were  used  as  the  basis  to  determine  an  appropriate  range  of  values  for 
the  seven  factors  selected.  The  NOLH  design  was  generated  using  a  range  of 
-30  to  30  for  each  factor.  These  values  are  meant  to  be  conservative  bounds 
that  represent  the  most  extreme  economic  situations  and  are  based  on  the 
findings  of  (Pinelis  &  Huff,  2014). 

E.  NOLH  DESIGN  GENERATION 

The  NOLH  design  was  generated  using  the  NOLH  worksheet  available  for 
download  from  the  SEED  Center  website  (https://harvest.nps.edu/software.htm). 
The  worksheet  calculates  the  factor  values  for  each  design  point  based  on  the 
ranges  selected.  The  NOLH  design  that  was  generated  consists  of  17  design 
points  for  seven  factors  and  is  able  to  fill  the  design  space  rather  well.  Figure  8 
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shows  a  pairwise  plot  that  displays  the  NOLH  design  projected  into  all  the  two- 
dimensional  subspaces.  Notice  how  the  data  points  are  distributed  and 
effectively  fill  the  design  space.  Moreover,  each  column  of  the  design  matrix  is 
nearly  orthogonal  to  the  others;  thereby,  guaranteeing  minimal  confounding 
between  estimates. 


Figure  8.  Scatterplot  Matrix  for  NOLH  Design 
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F.  IMPLEMENTATION  OF  NOLH  DESIGN 

This  study  relies  on  the  Navy  Total  Force  Strength  Management  System 

Graphical  User  Interface  for  the  implementation  and  execution  of  the  NOLH 

design,  and  also  for  the  collection  of  the  output  data  that  the  design  generates. 
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Currently,  the  only  way  to  implement  the  design  is  by  manually  creating  a  unique 
NTFSM  scenario  for  each  data  point  and  modifying  the  selected  factors 
according  to  their  designed  values.  This  process  is  extremely  time-consuming 
and  is  prone  to  mistakes,  so  extra  care  was  taken  to  ensure  the  integrity  of  the 
design  remained  intact.  The  implementation  of  a  design  of  experiments  is  better 
suited  to  be  handled  by  a  software  program;  however,  the  Navy  Total  Force 
Strength  Management  System  Graphical  User  Interface  does  not  currently  allow 
for  this.  It  is  recommended  that  this  capability  be  added  to  NTFSM  to  enable 
future  analysts  to  more  effectively  use  it. 

Utilizing  the  Navy  Total  Force  Strength  Management  System  Graphical 
User  Interface,  17  unique  NTFSM  scenarios,  one  each  for  every  design  point, 
were  created  and  titled  accordingly.  The  scenarios  were  initialized  to  use  fiscal 
year  2014  manpower  and  personnel  data.  NTFSM  was  run  to  project  one  fiscal 
year  into  the  future  (FY2015).  Fiscal  year  2014  data  was  selected  because  it  was 
the  most  recent  full  fiscal  year  data  available.  Once  the  scenarios  were  created 
and  initialized,  a  random  five-digit  seed  was  assigned  to  each  scenario.  The 
scenarios  were  then  run  30  times  each — for  a  total  of  510  NTFSM  test  runs.  The 
outputs  of  the  30  runs  were  then,  very  carefully,  copied  from  the  Uncertainty- 
Years  reports  that  were  generated  by  each  scenario  and  pasted  into  a 
spreadsheet.  The  spreadsheet  was  then  formatted  into  a  Comma  Separated 
Value  file  so  that  the  data  could  be  more  easily  transferred,  manipulated,  and 
analyzed. 

The  17  randomly  generated  five  digit  seed  values  were  populated  using  R, 
which  is  a  programing  language  commonly  used  for  statistical  computing 
(https://www.r-project.org/).  A  vector  consisting  of  the  numbers  0  through  9  was 
created  and  assigned  to  the  variable  “X.”  The  sample()  function  was  then  used  to 
randomly  select  five  numbers  from  that  vector  with  replacement.  Figure  9  shows 
the  R  script  which  was  used  as  well  as  an  example  of  a  five-digit  seed. 
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Figure  9.  R  code  Used  to  Generate  Random  Five-digit  Seeds 


>  #  create  vector  of  numbers  and  assign  to  variable  x 

>  x  <-  c(0 : 9) 

>  #  select  5  numbers  from  x  variable  with  replacement 

>  sampl e(x, 5 ,T) 

[1]  7  5  1  3  5 


G.  DATA  GENERATION  FOR  OUTPUT  DISTRIBUTION  ANALYSIS 

When  running  multiple  iterations  of  a  NTFSM  scenario,  the  Navy  Total 
Force  Strength  Management  System  Graphical  User  Interface  does  not  include 
the  outcome  of  every  simulation  run  in  any  of  its  reports.  This  information  could 
theoretically  be  collected  and  calculated  from  the  CSV  output  files  that  NTFSM 
generates  for  each  scenario;  however,  this  author,  as  well  as  the  SEED  Center 
technical  staff,  was  unable  to  successfully  download  these  files.  The  Uncertainty- 
Years  report  is  the  only  report  generated  by  NTFSM  that  gives  any  indication  of 
the  variability  of  a  scenario’s  output,  but,  although  this  is  useful  information,  it 
does  not  give  much  insight  on  the  distribution  of  NTFSM’s  output.  In  order  to  gain 
a  better  understanding  of  the  distributions  of  NTFSM’s  output,  a  set  of  100 
identical  scenarios  (other  than  the  random  number  seed)  was  created.  The 
scenarios  were  set  up  to  use  fiscal  year  2014  manpower  and  personnel  data  to 
project  one  year  into  the  future  (FY2015).  All  NTFSM  inputs  were  kept  at  their 
default  values.  A  random  five-digit  seed  was  generated  for  each  of  the  100 
scenarios  in  R,  using  the  before  mentioned  method.  The  outputs  of  the  100 
scenarios  were  then  manually  copied  from  the  Uncertainty-Years  reports  that 
were  generated  by  each  scenario  and  pasted  into  a  spreadsheet.  The 
spreadsheet  was  then  formatted  into  a  Comma  Separated  Value  file  so  that  the 
data  could  be  more  easily  transferred  and  analyzed. 
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IV.  ANALYSIS  AND  RESULTS 


This  chapter  describes  the  analysis  and  meta-modeling  of  the  NTFSM 
output  data  gathered  from  the  NOLH  design  of  experiments.  After  verifying 
NTFSM’s  ability  to  produce  repeatable  results,  and  assessing  the  stochastic 
variation  and  distribution  of  NTFSM,  the  mean  values  of  the  design  point  outputs 
are  used  as  observations  to  build  a  set  of  Ordinary  Least  Squares  (OLS) 
regression  models.  These  models  give  insight  into  the  underlying  processes 
inherent  to  NTFSM  and  the  effects  that  the  explored  factors  have  on  NTFSM’s 
output. 

A.  ANALYTICAL  TOOLS 

Collection  and  organization  of  the  data  was  accomplished  using  Microsoft 
Excel  2010.  The  data  was  then  formatted  as  CSV  files  in  order  be  more  easily 
saved  and  transferred.  Analysis  and  meta-modeling  of  the  resulting  CSV  data 
files  were  performed  using  JMP  Pro  version  1 1 .0.0. 2 

B.  ASSESSMENT  OF  NTFSM’S  ABILITY  TO  PRODUCE  REPEATABLE 

RESULTS 

It  is  extremely  important  to  ensure  that  NTFSM’s  output  is  repeatable 
before  NTFSM  can  leave  the  testing  phase.  The  repeatability  of  NTFSM  output  is 
a  big  concern  for  N100.  The  manpower  and  personnel  forecasts  and  analyses 
that  N100  generates  are  used  by  top-level  decision  makers  when  considering 
changes  to  Navy-wide  manpower  and  personnel  policies  and  must  be  able  to 
stand  up  to  extreme  scrutiny.  Therefore,  independent  verification  of  results  by 
multiple  manpower  and  personnel  analysts  is  necessary.  This  independent 
verification  cannot  be  accomplished  unless  NTFSM  output  can  be  repeated. 


2  More  information  about  JMP  Pro  statistical  software  can  be  found  on  their  website  at 
http://www.jmp.com/en_us/home.html. 
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1 .  Single  Run  Output  Repeatability  Assessment 

To  test  NTFSM’s  ability  to  repeat  the  results  of  a  single  simulation  run  of  a 
scenario,  two  identical  NTFSM  scenarios  were  created.  The  scenarios  were  set 
up  to  run  one  iteration  of  the  scenario  and  use  fiscal  year  2014  manpower  and 
personnel  data  to  project  one  year  into  the  future  (FY2015).  All  scenario  inputs 
were  kept  at  their  default  values  and  each  scenario  used  the  same  five  digit 
random  seed  (41701),  which  was  generated  using  the  R  coding  language.  The 
Monthly  Summary  reports  generated  by  each  scenario  were  then  visually 
inspected  for  any  differences.  No  differences  in  the  Monthly  Summary  reports 
were  observed.  That  is,  the  two  scenarios  precisely  repeated  each  other’s 
results.  This  experiment  was  conducted  a  second  time  and  the  seed  used  was 
modified  (66389).  The  results  of  this  experiment  were  consistent  with  the  first, 
i.e.,  both  scenarios,  using  the  same  seed,  generated  identical  Monthly  Summary 
reports.  It  was  also  observed  that  the  Monthly  Summary  reports  generated  in  the 
first  experiment  varied  greatly  from  the  Monthly  Summary  reports  generated  by 
the  second  experiment  even  though  the  only  difference  between  the  two 
experiments  was  the  seeds  used.  This  result  proves  that  NTFSM  output  can 
depend  of  the  random  seed  chosen.  Appendix  B  contains  the  Monthly  Summary 
reports  generated  by  these  two  experiments. 

2.  Multiple  Run  Output  Repeatability  Assessment 

To  test  NTFSM’s  ability  to  repeat  the  results  of  multiple  simulation  runs  of 
a  scenario,  two  identical  NTFSM  scenarios  were  created.  The  scenarios  were  set 
up  to  simulate  five  iterations  of  the  scenario  and  use  fiscal  year  2014  manpower 
and  personnel  data  to  project  one  year  into  the  future  (FY2015).  All  scenario 
inputs  were  kept  at  their  default  values  and  each  scenario  used  the  same  five 
digit  random  seed  (37295),  which  was  generated  using  the  R  coding  language. 
The  Monthly  Summary  reports  generated  by  each  scenario  were  then  visually 
inspected  for  any  differences.  No  differences  in  the  Monthly  Summary  reports 
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were  observed.  The  two  scenarios  repeated  each  other’s  results.  Appendix  B 
contains  the  Monthly  Summary  reports  generated  by  this  experiment. 

C.  SELECTION  OF  THE  NTFSM  OUTPUTS  TO  BE  ANALYZED  AND 

USED  AS  RESPONSES  FOR  META-MODELS 

Since  the  Navy  Total  Force  Management  System  Graphical  User  Interface 
only  produces  a  measure  of  variability  for  the  outputs  listed  on  the  Uncertainty- 
Years  report,  these  were  the  only  NTFSM  outputs  considered  for  analysis  and 
meta-modeling.  This  study  focuses  on  the  NTFSM  outputs  listed  on  the 
Uncertainty-Years  report  which  pertain  to  End  Strength,  to  include  enlisted  losses 
and  enlisted  gains.  The  specific  outputs  selected  for  analysis  and  meta-model 
responses  are  listed  in  Table  2. 


Table  2.  List  of  NTFSM  Outputs  Selected  for  Analysis  and 
Meta-Model  Responses 


1 

2 

3 

4 

5 

6 
7 


D.  DISTRIBUTION  ANALYSIS  OF  NTFSM  OUTPUT 

When  running  multiple  iterations  of  a  NTFSM  scenario,  the  Navy  Total 
Force  Strength  Management  System  Graphical  User  Interface  does  not  include 
much  information  about  the  distribution  of  outputs  in  any  of  its  reports.  To  gain  a 
better  understanding  of  the  distribution  of  NTFSM’s  output  and  the  behavior  of 
the  model  as  a  whole,  100  independent  but  identical  (other  than  the  random 
number  seeds)  scenarios  were  manually  generated.  An  analysis  of  the 
distribution  of  the  selected  output  of  these  100  scenarios  was  conducted  and  it 
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Outputs 

End  Strength 
Attrition  Losses 
Retirement  Losses 
Recruit  Losses 
EAOS  Losses 
Prior  Service  Gains 
Recruit  Gains 


was  found  that  all  of  the  selected  outputs  with  the  exception  of  End  Strength 
were  approximately  normally  distributed.  A  detailed  analysis  of  the  distribution  of 
Attrition  Losses  and  End  Strength  follows.  A  summary  of  the  distribution  of 
Retirement  Losses,  Recruit  Losses,  EAOS  Losses,  Prior  Service  Gains,  and 
Recruit  Gains  can  be  found  in  Appendix  C. 

1 .  Analysis  of  the  Distribution  of  the  Attrition  Losses  Output  Data 
a.  Descriptive  Statistics 

Descriptive  statistics  for  100  independent  observations  of  the  Attrition 
Losses  output,  which  are  in  units  of  enlisted  Sailors,  were  calculated,  the 
observations  ranged  from  9849  to  10312.  Other  relevant  descriptive  statistics  are 
summarized  in  Figure  10. 


Figure  10.  Summary  of  Descriptive  Statistics  for  Attrition  Losses  Output 


Summary  Statistics 

10033.270 
97.828 
9.783 
10052.681 
10013.859 
100.000 
9849.000 
10312.000 
10034.500 


Mean 
Std  Dev 
Std  Err  Mean 
Upper  95%  Mean 
Lower  95%  Mean 
N 

Minimum 

Maximum 

Median 


b.  Distribution  Fitting 

A  distribution  was  fit,  and  it  was  found  that  the  data  are  approximately 
normally  distributed  with  an  estimated  mean  of  10033.27  and  an  estimated 
standard  deviation  of  97.83.  These  parameter  estimates,  as  well  as  their  upper 
and  lower  95%  confidence  intervals,  are  shown  in  Figure  1 1 . 
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Figure  1 1 .  Parameter  Estimates  for  Normal  Distribution  Fit  to  Attrition 

Losses  Data 

Parameter  Estimates 

Type  Parameter  Estimate  Lower  95%  Upper  95% 

Location  \x  10033.270  10013.859  10052.681 

Dispersion  o  97.828  85.894  113.645 


c.  Goodness-of-Fit  Testing 

In  order  to  gain  a  better  understanding  to  how  well  the  Attrition  Losses 
data  matches  a  normal  distribution,  a  histogram,  box  plot,  and  normal  quantile 
plot  were  generated  for  the  data  and  compared  to  a  histogram,  box  plot,  and 
normal  quantile  plot  that  was  generated  using  100  standard  normal  observations. 
Normal  data  tend  to  stay  on  the  diagonal  red  line  shown  on  a  normal  quantile  plot 
and  histograms  of  normal  data  tend  to  appear  bell  shaped  in  form.  No  indication 
of  a  reasonable  difference  between  the  two  sets  of  plots  is  visually  apparent 
when  compared  side-by-side.  Figure  12  shows  a  side-by-side  comparison  of  a 
histogram,  box  plot,  and  normal  quantile  plot  for  the  Attrition  Losses  data  (left) 
and  100  observations  generated  from  of  a  standard  normal  distribution  (right). 
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Figure  12.  Side-by-side  Comparison  of  a  Histogram,  Box  Plot,  and 

Normal  Quantile  Plot  for  the  Attrition  Losses  Output  Data  (left) 
and  100  Observations  Generated  from  a  Standard  Normal 
Distribution  (right) 


Normal  Quantile  Plots  appear  at  the  top,  Box  Plots  are  shown  in  the  center,  and 
Histograms  are  shown  on  the  bottom,  of  the  two  charts. 


In  order  to  conduct  a  more  quantitative  goodness-of-fit  test  a  Shapiro-Wilk 
goodness-of-fit  test  was  performed.  The  Shapiro-Wilk  goodness-of-fit  test 
assesses  the  null  hypothesis  that  the  data  are  from  a  normal  distribution. 
Statistical  standards  usually  require  for  the  null  to  be  rejected  at  a  p-value  of  less 
than  0.05.  A  p-value  of  .184  was  obtained  from  the  Shapiro-Wilks  goodness-of-fit 
test  performed  on  the  Attrition  Losses  data;  therefore,  the  null  hypothesis  was  not 
rejected  and  it  can  be  reasonable  determined  that  the  Attrition  Losses  output 
data  are  approximately  normally  distributed.  A  summary  of  the  results  of  the 
Shapiro-Wilk  goodness-of-fit  test  conducted  on  the  Attrition  Losses  data  is  shown 
in  Figure  13. 
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Figure  13.  Summary  of  the  Results  of  the  Shapiro-Wilk  goodness-of-Fit 
Test  Conducted  on  the  Normal  Distribution  Fitted  to  the  Attrition 

Losses  Data 


P-value  is  indicated  by  the  value  listed  as  Prob<W. 

2.  Analysis  of  the  Distribution  of  the  End  Strength  Output  Data 
a.  Descriptive  Statistics 

Descriptive  statistics  for  100  independent  observations  of  the  End 
Strength  output,  which  are  in  units  of  Enlisted  Sailors,  were  calculated;  the 
observations  ranged  from  265266  to  265288.  The  estimated  standard  error  for 
End  Strength  (3.8  Enlisted  Sailors)  is  several  orders  of  magnitude  smaller  than 
the  estimated  mean.  This  indicates  that  there  is  a  very  small  amount  of 
stochastic  variability  in  NTFSM’s  End  Strength  output.  Other  relevant  descriptive 
statistics  are  summarized  in  Figure  14. 


Figure  14.  Summary  of  Descriptive  Statistics  for  End  Strength  Output 


Summary  Statistics 

Mean 

265277.04 

Std  Dev 

3.848 

Std  Err  Mean 

0.385 

Upper  95%  Mean 

265277.80 

Lower  95%  Mean 

265276.28 

N 

100.000 

Minimum 

265266.00 

Maximum 

265288.00 

Median 

265277.00 
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b.  Distribution  Fitting 

An  attempt  was  made  to  fit  a  distribution  to  the  End  Strength  output  data, 
and,  unlike  the  other  selected  outputs,  it  was  found  that  the  data  were  not 
normally  distributed.  A  Shapiro-Wilk  goodness-of-fit  test  that  was  performed  on  a 
normal  distribution  that  had  been  fit  to  the  data  resulted  in  a  p-value  of  0.0215. 
This  results  in  the  rejection  of  the  null  hypothesis  that  the  data  are  normally 
distributed  at  the  commonly  used  .05  significance  level.  Several  other  common 
distributions  were  also  fit  to  the  data,  including  a  gamma,  Weibull,  exponential, 
log  normal,  normal  2  mixture,  and  generalized  logarithm  distribution,  none  of 
which,  however,  had  a  Shapiro-Wilk  p-value  greater  than  0.05.  The  End  Strength 
output  data  does  not  seem  to  fit  any  of  the  commonly  used  statistical 
distributions.  A  histogram,  box  plot,  and  normal  quantile  plot  of  the  End  Strength 
data  are  shown  in  Figure  15.  One  can  observe  from  the  quantile  plot  that  with 
such  a  short  range  of  output,  the  discrete  nature  of  the  response  makes  it 
significantly  different  than  a  continuous  normal.  A  summary  of  the  parameter 
estimates  and  results  of  the  Shapiro-Wilk  goodness-of-Fit  test  for  a  normal 
distribution  that  was  fit  to  the  data  is  shown  in  Figure  16. 
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Figure  15.  Quantile  Plot  Histogram,  Box  Plot,  and  Normal  Quantile  Plot 

for  the  End  Strength  Output  Data 


Figure  16.  Parameter  Estimates  and  Summary  of  the  Results  of  the 
Shapiro-Wilk  Goodness-of-Fit  Test  fora  Normal  Distribution  Fit 
to  the  End  Strength  Data 

Parameter  Estimates 

Type  Parameter  Estimate  Lower  95% 

Location  p  265277.04  265276.28 

Dispersion  a  3.848  3.378 

-2log(Likelihood)  =  552.294095224667 

Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test 

W  Prob<W 

0.970  0.021 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject  Ho. 


P-value  is  indicated  by  the  value  listed  as  Prob<W. 


Upper  95% 

265277.80 

4.470 
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E.  INITIAL  ASSESSMENT  OF  NOLH  DESIGN  OF  EXPERIMENTS 

OUTPUT  DATA 

After  reviewing  the  output  data  from  the  Uncertainty-Years  reports 
generated  from  the  design  of  experiments,  it  was  noticed  that  the  output  data  for 
enlisted  Officer  Candidates,  Start,  Graduation  and  Failures,  had  values  of  zero 
for  each  of  the  17  design  points.  After  reviewing  all  available  NTFSM  documents, 
this  author  could  not  find  an  explanation  for  this  result  but  speculates  that  either: 
(1 )  Officer  Candidate  data  was  not  contained  in  the  fiscal  year  2014  data  used  by 
the  scenarios,  (2)  Officer  Candidate  data  must  be  entered  into  NTFSM  by  the 
user  prior  to  running  the  simulation,  or,  (3)  because  NTFSM  is  still  in  the  testing 
phase,  its  capability  to  track  Officer  Candidates  is  still  in  development.  It  was  also 
noticed  that  there  was  an  extremely  small  amount  of  stochastic  variation  in  the 
End  Strength  output.  The  estimated  mean  End  Strength  results  of  all  17  design 
points  ranged  from  265,276  to  265,279 — which  is  a  difference  of  only  three,  this 
seems  like  a  very  small  amount  of  variation  given  the  wide  range  of  factor  values 
explored  in  the  design. 

F.  ANALYSIS  AND  METAMODELING  OF  OUTPUT  GENERATED  BY  THE 

NOLH  DESIGN  OF  EXPERIMENTS 

Meta-models  help  provide  insight  on  the  underlying  system  of  processes 
inherent  to  a  simulation  by  providing  information  on  which,  and  to  what  extent, 
simulation  inputs  affect  simulation  outputs.  By  using  regression  techniques  that 
use  simulation  outputs  as  response  variables  and  simulation  inputs  (factors)  as 
prediction  variables,  meta-models  produce  estimated  coefficient  values  that 
quantify  how  simulation  factors  affect  simulation  output.  This  section  provides  a 
description  of  the  seven  meta-models  (one  for  each  of  the  selected  NTFSM 
outputs)  that  were  analyzed  by  this  study. 

1.  Meta-Modeling  Methodology 

Seven  independent  stepwise  regressions  were  conducted,  each  one 
utilizing  a  different  NTFSM  output  as  the  response  variable.  A  common  approach 
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to  building  linear  models  is  to  start  with  a  wide  scope  and  include  all  predictor 
variables  as  well  as  possible  interaction  and  nonlinear  terms  (Crawley,  2013).  In 
this  study,  all  initial  stepwise  regression  models  include  main  effects  consisting  of 
the  seven  NTFSM  economic  coefficients  explored,  all  two-way  and  three-way 
interactions,  and  3rd  order  polynomials.  This  helps  ensure  the  meta-model’s 
ability  to  capture  any  interactions  or  non-linearity  in  the  data.  A  minimum 
Bayesian  information  criterion  (BIC)  stopping  rule  was  used  to  help  select  which 
terms  should  be  utilized  by  the  meta-model;  this  helps  prevent  overfitting  of  the 
model.  Stepwise  regression  is  an  approach  that  is  used  for  selecting  a  subset  of 
effects  for  a  regression  model.  It  is  used  when  there  is  little  theory  to  guide  the 
selection  of  terms  for  a  model  and  the  modeler  wants  to  use  what  seems  to 
provide  a  good  fit  (SAS  Institute  Inc.,  2013).  Stepwise  regression  computes 
estimates  that  are  the  same  as  those  of  other  least  squares  platforms,  but  it 
facilitates  searching  and  selecting  among  many  models  (SAS  Institute  Inc., 
2013).  Additional  information  on  the  Bayesian  information  criterion,  stepwise 
regressions,  and  linear  regressions  can  be  found  in  SAS  Institute  Inc.  (2013). 

The  terms  which  were  selected  with  the  help  of  stepwise  regression  were 
then  used  as  prediction  variables  in  a  least  squares  regression  model.  The  least 
squares  regression  models’  R-square  and  adjusted  R-square  values  were 
screened  to  verify  the  model’s  performance.  The  R-square  value  is  the  proportion 
of  the  response  variance  explained  by  the  input  variables,  and  ranges  from  zero 
to  one.  The  R-square  and  Adjusted  R-square  values  from  the  accepted  models 
are  listed  in  Table  3.  All  seven  fitted  regression  models  explained  well  over  90 
percent  of  the  responses’  variability. 
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Table  3.  R-Square  and  Adjusted  R-Square  of  Accepted  Meta- 

Models 

_ R2  Adj  R2 


End  Strength 

0.910 

0.840 

Attrition  Losses 

0.999 

0.999 

Retirement  Losses 

0.947 

0.94 

Recruit  Losses 

0.989 

0.984 

EAOS  Losses 

0.959 

0.953 

Prior  Service  Gains 

0.875 

0.778 

Recruit  Losses 

0.983 

0.979 

The  least  squares  models  were  then  validated  by  testing  key  model 
assumptions  before  being  accepted.  Verification  of  the  accepted  models  was 
conducted  by  comparing  the  observed  output  from  three  independent  NTFSM 
Test  scenarios,  which  were  created,  to  the  predicted  values  produced  by  the 
accepted  meta-models. 

2.  Overview  of  the  Meta-Model  that  Was  Selected  for  End 
Strength 

For  the  End  Strength  output,  a  stepwise  regression  model  of  all  effects 
and  three  way  interactions  and  third  degree  polynomials  produced  a  model  with 
15  predictor  variables  and  had  R-square  and  Adjusted  R-square  values  of  1 .0000 
and  0.9997  respectively.  This  model  contained  five  of  the  seven  NTFSM 
economic  coefficients,  five  interaction  terms  and  five  polynomial  terms.  This 
shows  that  complicated  relationships  are  captured  by  NTFSM.  Many  of  the 
interaction  and  polynomial  terms  contributed  very  little  to  the  R-square  and 
Adjusted  R-square  values  and/or  had  high  t-test  p-values.  It  was  determined  that 
these  interaction  and  polynomial  terms  did  not  add  sufficient  value  to  the  model 
and  were  removed,  for  parsimony.  The  remaining  terms,  however,  seem  to  have 
very  small  coefficient  values  given  that  Navy  End  Strength  is  generally  in  the 
hundreds  of  thousands.  The  final  model  is  summarized  in  Figure  17. 
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Figure  17.  Parameter  Estimates  for  End  Strength  Meta-Model 


Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

265277.47 

0.139898 

1.9e+6 

<.0001 

Reenlist  P 

-0.034173 

0.005407 

-6.32 

0.0001 

Long  Ext  U 

0.0186663 

0.005635 

3.31 

0.0090 

Recruit  P 

-0.011765 

0.004891 

-2.41 

0.0396 

Attrite  U 

0.0750909 

0.014977 

5.01 

0.0007 

Reenlist  P*Long  Ext  U 

0.0016202 

0.000294 

5.51 

0.0004 

Attrite  U*Attrite  U 

-0.001038 

0.000318 

-3.27 

0.0097 

Attrite  U*Attrite  U*Attrite  U 

-0.000101 

2.341e-5 

-4.34 

0.0019 

The  R-square  and  Adjusted  R-square  of  0.9102  and  0.8404,  respectively, 
show  that  the  model  sufficiently  accounts  for  the  variability  of  the  data,  however 
the  model  utilizes  eight  parameters  and  since  there  are  only  17  data  points,  this 
model  runs  the  risk  of  over-fitting  the  data.  Inspection  of  the  t-test  p-values  shows 
that  of  the  parameters  used  are  highly  significant  and  therefore  all  eight 
parameters  are  kept  in  the  model.  Based  on  the  coefficients,  the  Attrition  Losses 
Unemployment,  Reenlistment  Pay,  and  Long  Extension  Unemployment, 
economic  factors  have  the  greatest  effect  on  NTFSM’s  End  Strength  output.  The 
effects  these  economic  factors  have  on  the  End  Strength  output  are  still 
extremely  small  when  compared  to  the  intercept  estimate  of  approximately 
265,278.  An  R-square  and  Adjusted  R-square  of  0.34  and  0.188,  respectively, 
were  obtained  from  a  model  built  with  only  these  three  predictors,  which  indicates 
that  the  other  terms  in  the  model  still  have  some  effect. 

Diagnostic  plots  of  the  model  indicate  that  key  modeling  assumptions  are 
met.  The  residual  versus  predicted  plot  in  Figure  18  indicates  homoscedasticity 
of  the  residuals  and  the  normal  quantile  plot  of  the  residuals  shown  in  Figure  19 
exhibits  behavior  consistent  with  normally  distributed  data — though  further 
investigation  into  the  striped  pattern  is  warranted.  For  further  verification,  a 
Shapiro-Wilk  goodness-of-fit  test  was  conducted  on  a  normal  distribution  fit  to  the 
residuals  of  the  End  Strength  meta-model.  Parameter  estimates  and  goodness- 

of-fit  statistics  are  summarized  in  Figure  20. 
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Figure  18.  Residuals  versus  Predicted  Values  of  End  Strength  Meta- 

Model 


Residual  by  Predicted  Plot 


Figure  19.  Side-by-Side  Comparison  of  a  Normal  Quantile  Plot  of  the 
Residuals  for  End  Strength  Meta-Model  (Right)  and  Normal 
Quantile  Plot  for  Normal  Data  (Left) 
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Figure  20.  Parameter  Estimates  and  Goodness-of-Fit  Statistics  for 
Normal  Distribution  Fit  to  the  End  Strength  Meta-Model 

Residuals 


Parameter  Estimates 

Type  Parameter  Estimate  Lower  95%  Upper  95% 


Location 

F 

2.739e-ll 

-0.142864 

0.1428641 

Dispersion 

a 

0.2778633 

0.2069442 

0.4228882 

-2log(Likelihood)  =  3.70262463835185 

Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test 

W  Prob<W 

0.985792  0.9918 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject  Ho. 


3.  Overview  of  the  Meta-Model  that  Was  Selected  for  Attrition 
Losses 

For  the  Attrition  Losses  output,  a  stepwise  regression  model  of  all  effects 
and  three  way  interactions  and  third  order  polynomials  produced  a  model  with  15 
predictor  variables  that  had  R-square  and  Adjusted  R-square  values  of  1 .0000 
and  1.0000,  respectively.  The  extremely  high  (i.e.,  perfect  fit)  R-square  and 
Adjusted  R-square  values  give  indication  that  the  model  has  been  over-fit,  so 
further  analysis  of  the  model  terms  was  conducted.  The  initial  model  contained 
five  of  the  seven  NTFSM  economic  factors,  five  interaction  terms  and  five 
polynomial  terms.  It  was  found  that  many  of  the  terms  contributed  very  little  to  the 
R-square  and  Adjusted  R-square  values.  A  model  was  created  that  used  only  the 
Attrition  Losses  Unemployment  and  Pay  economic  factors;  this  model  produced 
an  R-square  and  adjusted  R-square  of  0.9996  and  0.9994,  respectively.  Again, 
almost  a  perfect  fit.  When  validation  of  this  model  was  performed,  however,  it 
was  found  that  the  residuals  were  not  normally  distributed.  The  final  model  which 
was  selected  utilized  the  Attrition  Losses  Unemployment  as  well  as  the  Attrition 
Losses  Pay  economic  factors  and  their  corresponding  2nd  and  3rd  order 
polynomial  terms.  The  final  model  is  summarized  in  Figure  21 . 
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Figure  21 .  Parameter  Estimates  for  Attrition  Losses  Meta-Model 


Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

9774.4582 

147.3555 

66.33 

<.0001 

Attrite  U 

-608.2732 

14.94277 

-40.71 

<.0001* 

Attrite  P 

3.5402206 

6.003497 

0.59 

0.5663 

Attrite  U*Attrite  U 

20.76605 

0.326185 

63.66 

<.0001* 

Attrite  U*Attrite  U*Attrite  U 

-0.33099 

0.023085 

-14.34 

<.0001* 

The  R-square  and  Adjusted  R-square  of  0.9996  and  0.9994,  respectively, 
show  that  the  model  does  a  very  good  job  of  accounting  for  the  variation  of  the 
response  variable.  Based  on  the  coefficients,  the  Attrition  Losses  Unemployment, 
economic  factor  has  the  greatest  effect  on  NTFSM’s  Attrition  Losses  output.  An 
R-square  and  Adjusted  R-square  of  0.84  and  0.83,  respectively,  were  obtained 
from  a  model  built  using  only  the  Attrition  Losses  Unemployment  economic 
factor,  which  indicates  that  this  one  term  explains  most  of  the  variation  in  the 
model,  but  the  other  terms  still  have  a  significant  effect.  All  terms  except  Attrition 
Losses  Pay  are  highly  statistically  significant  according  to  their  t-test  p-values.  A 
model  that  excludes  the  Attrition  Losses  Pay  term  shows  signs  of  non-normality 
in  the  residuals  and  therefore  Attrition  Losses  Pay  was  kept  in  the  model. 

Diagnostic  plots  of  the  final  model  indicate  that  key  modeling  assumptions 
are  met.  The  residual  versus  predicted  plot  in  Figure  22  sufficiently  indicates 
homoscedasticity  of  the  residuals  and  the  normal  quantile  plot  of  the  residuals 
shown  in  Figure  23  exhibits  behavior  consistent  with  normally  distributed  data. 
For  further  verification,  a  Shapiro-Wilk  goodness-of-fit  test  was  conducted  on  a 
normal  distribution  fit  to  the  residuals  of  the  Attrition  Losses  meta-model. 
Parameter  estimates  and  goodness-of-fit  statistics  are  summarized  in  Figure  24. 
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Residuals  Versus  Predicted  Values  of  Attrition  Losses  Meta-Model 


Figure  22.  Side-by-side  Comparison  of  a  Normal  Quantile  Plot  of  the 
Residuals  for  Attrition  Losses  Meta-Model  (Right)  and  Normal 
Quantile  Plot  for  Normal  Data  (Left) 


43 


Figure  23.  Parameter  Estimates  and  Goodness-of-Fit  Statistics  for 
Normal  Distribution  Fit  to  the  Attrition  Losses  Meta-Model 

Residuals 

Parameter  Estimates 

Type  Parameter  Estimate 

Location  |i  -4.28e-13 

Dispersion  o  349.755 

-2log(Likelihood)  =  246.38982867376 

Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test 

W  Prob<W 

0.907858  0.0920 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject  Ho. 


Lower  95%  Upper  95% 

-179.8274  179.82743 

260.48697  532.30225 


4.  Overview  of  the  Meta-Model  that  Was  Selected  for  EAOS 
Losses 

For  the  EAOS  Losses  output,  a  stepwise  regression  model  of  all  effects 
and  three  way  interactions  and  third  order  polynomials  produced  a  model  with  15 
predictor  variables  that  had  R-square  and  Adjusted  R-square  values  of  1 .0000 
and  1.0000  respectively.  Again,  a  perfect  fit.  The  extremely  high  R-square  and 
Adjusted  R-square  values  obtained  give  indication  that  the  model  has  been  over 
fit,  therefore  further  analysis  of  the  model  terms  was  conducted.  The  initial  model 
contained  six  of  the  seven  NTFSM  economic  factors,  four  interaction  terms  and 
five  polynomial  terms.  It  was  found  that  many  of  the  terms  contributed  very  little 
to  the  R-square  and  Adjusted  R-square  values  and  by  using  only  two  terms, 
Reenlistment  Unemployment  and  Long  Extension  Unemployment,  a  model 
producing  an  R-square  and  Adjusted  R-square  of  0.9591  and  0.9533, 
respectively,  was  obtained.  The  final  model  is  summarized  in  Figure  25. 


44 


Figure  24.  Parameter  Estimates  for  EAOS  Losses  Meta-Model 


Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

12946.1 

350.48 

36.94 

<.0001 

Reenlist  U 

325.512 

19.077 

17.06 

<.0001 

Long  Ext  U 

116.361 

19.077 

6.10 

<.0001 

The  R-square  and  Adjusted  R-square  of  0.959  and  0.953  respectively, 
show  that  the  model  does  a  very  good  job  of  accounting  for  the  variation  of  the 
response  variable.  Based  on  the  coefficients,  the  Reenlistment  Unemployment, 
economic  factor  has  the  greatest  effect  on  NTFSM’s  Attrition  Losses  output. 
Notice  that  the  Reenlistment  Pay  economic  factor  does  not  enter  the  model, 
therefore  it  can  be  reasonably  concluded  that  the  effect  that  the  Reenlistment 
Pay  economic  coefficient  has  on  the  EAOS  Losses  output  is  negligible. 

Diagnostic  plots  of  the  final  model  indicate  that  key  modeling  assumptions 
are  met.  The  residual  versus  predicted  plot  in  Figure  26  sufficiently  indicates 
homoscedasticity  of  the  residuals  and  the  normal  quantile  plot  of  the  residuals 
shown  in  Figure  27  exhibits  behavior  consistent  with  normally  distributed  data. 
For  further  verification,  a  Shapiro-Wilk  goodness-of-fit  test  was  conducted  on  a 
normal  distribution  fit  to  the  residuals  of  the  EAOS  Losses  meta-model. 
Parameter  estimates  and  goodness-of-fit  statistics  are  summarized  in  Figure  28. 


Figure  25.  Residuals  versus  Predicted  Values  of  EAOS  Losses 

Residual  by  Predicted  Plot 
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Figure  26.  Side-by-side  Comparison  of  a  Normal  Quantile  Plot  of  the 
Residuals  for  the  EAOS  Losses  Meta-Model  (Right)  and 
Normal  Quantile  Plot  for  Normal  Data  (Left) 


Figure  27.  Parameter  Estimates  and  Goodness-of-Fit  Statistics  for 
Normal  Distribution  Fit  to  the  EAOS  Losses  Meta-Model 

Residuals 


Parameter  Estimates 

Type 

Parameter 

Estimate 

Lower  95% 

Upper  95% 

Location 

F 

-le-12 

-694.989 

694.989 

Dispersion 

o 

1351.72 

1006.719 

2057.218 

-2log(Likelihood)  =  292.354380166014 

Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test 

W  Prob<W 

0.953  0.504 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject  Ho. 
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5. 


Comparison  of  Prediction  Estimates  and  Observed  NTFSM 
Output 


To  verify  the  meta-models  that  were  developed,  three  independent 
NTFSM  scenarios  were  run  to  test  the  meta-models’  predictive  accuracy.  The 
first  test  scenario  set  all  seven  of  the  NTFSM  economic  coefficients  explored  to 
the  upper  bound  of  30.  The  second  test  scenario  set  all  seven  of  the  NTFSM 
economic  coefficients  explored  to  the  lower  bound  of  -30.  The  third  test  scenario 
set  each  of  the  seven  NTFSM  economic  coefficients  explored  to  a  uniformly 
distributed  random  number  between  -30  and  30.  The  outputs  of  these  scenarios 
were  compared  to  the  prediction  values  produced  by  the  meta-models.  The 
NTFSM  economic  coefficient  values  used  in  the  third  test  scenario  are  listed  in 
Table  4. 


Table  4.  NTFSM  Economic  Coefficient  Values  Used  in  the 

Third  Test  Scenario 


Event 

Unemployment 

Pay 

Reenlistment 

19.61 

4.44 

Long  Extensions: 

18.54 

0.00 

PS  Gains: 

0.00 

0.00 

Recruit  Losses: 

-17.94 

5.42 

Attrition  Losses: 

3.04 

17.10 

Retirement  Losses: 

0.00 

0.00 

It  was  previously  determined  that  the  NTFSM  outputs  explored  by  this 

thesis  are  approximately  normally  distributed,  with  the  exception  of  the  End 

Strength  output.  It  was  therefore  possible  to  calculate  95  percent  upper  and 

lower  confidence  bounds  using  the  estimated  means  and  standard  errors  listed 

on  the  Uncertainty-Years  reports  of  the  three  test  scenarios  described  in  this 

section.  The  distribution  of  the  End  Strength  output  is  unknown,  therefore  95 

percent  confidence  bounds  could  not  be  calculated;  instead  it  was  determined 

sufficient  to  set  the  bounds  to  plus  or  minus  one  standard  error  from  the 

estimated  mean.  Comparison  plots  of  the  prediction  estimates  and  observed 

NTFSM  output  values  of  the  End  Strength  output  for  each  of  the  three  test 
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scenarios  are  shown  in  Figure  29.  Comparison  plots  for  the  remaining  meta¬ 
models  can  be  found  in  Appendix  E. 


Figure  28.  Comparison  Plots  for  End  Strength  Output 
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6.  Analysis  of  the  Variance  of  the  End  Strength  Output 

Visual  inspection  of  the  comparison  plots  shown  in  Figure  29  and  of  the 
data  generated  by  the  NOLH  design  of  experiments  show  very  little  difference  in 
the  observed  NTFSM  End  Strength  output  values.  To  gain  a  better 
understanding  of  the  differences  in  the  End  Strength  output  values,  a  side-by- 
side  comparison  plot  was  created.  The  comparison  plot  shows  the  End  Strength 
output  values  with  upper  and  lower  bounds  of  plus  or  minus  one  standard  error, 
for  each  of  the  17  design  points  of  the  NOLH  design  of  experiments,  the  upper 
and  lower  bounds  testing  scenarios,  and  two  new  extreme  scenarios  which  set 
the  seven  NTFSM  economic  coefficients  to  the  minimum  and  maximum  values 
allowed  by  the  NTFSM  software  (-999.99  and  9999.99).  It  was  found  that  the 

NTFSM  End  Strength  output  from  these  scenarios  are  all  well  within  one 
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standard  error  of  the  grand  mean  of  265,277,  therefore  it  can  be  reasonably 
concluded  that,  even  though  there  is  an  extreme  difference  in  the  input  values 
used  for  the  scenarios,  there  is  no  practical  difference  between  the  scenarios’ 
End  Strength  values.  Figure  30  shows  the  comparison  plot  which  was  created. 


Figure  29.  Comparison  Plot  of  End  Strength  Values  of  DOE  Design 
Points,  Upper  and  Lower  Bounds,  and  Extreme  Upper  and 
Lower  Bounds  Scenarios 
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G.  ASSESSMENT  OF  NTFSM’S  RUN  TIME 

NTFSM  is  currently  housed  on  an  NMPBS  testing  server.  Simulation  run 
time  varies  depending  on  a  number  of  variables,  including  server  traffic  and  the 
number  of  fiscal  years  that  are  being  simulated.  The  NTFSM  simulations  in  this 
study  were  run  for  a  single  fiscal  year.  In  the  process  of  running  the  experiments 
necessary  to  answer  the  research  questions  that  guide  this  study,  the  amount  of 
time  it  took  to  complete  a  single  NTFSM  simulation  run,  as  well  as  the  amount  of 
time  it  took  to  complete  30  runs  of  one  experimental  design  point,  were  recorded 
and  assessed. 

1.  Single  Fiscal  Year  NTFSM  Scenario  Run  Time 

In  order  to  better  understand  the  distribution  of  NTFSM’s  outputs,  100 
identical  scenarios  were  created.  The  scenarios  used  unique  five-digit  seeds  and 
were  initialized  to  use  fiscal  year  2014  Manpower  and  Personnel  data  to  project 
one  year  into  the  future  (FY2015).  The  time  to  run  one  scenario  ranged  from  4.65 
to  28.28  minutes,  with  an  average  run  time  of  approximately  eight  minutes. 
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These  and  other  descriptive  statistics,  as  well  as  a  histogram  and  box  plot,  of  the 
run  time  of  a  single  iteration  of  a  NTFSM  scenario  which  projects  over  a  time- 
horizon  consisting  of  a  single  fiscal  year  is  summarized  in  Figure  31 . 


Figure  30.  Histogram,  Box  Plot,  and  Descriptive  Statistics  of  the  Run 
Time  of  a  Single  Iteration  of  a  NTFSM  Scenario  that  Projects 
over  a  Time-Horizon  Consisting  of  a  Single  Fiscal  Year 


7.9854 

4.2481336 

0.4248134 

8.8283219 

7.1424781 

100 

4.65 

28.28 

6.625 


Horizontal  axis  shows  Total  Run  Time,  units  are  in  minutes. 


2.  NOLH  Design  of  Experiments  Run  Time  per  Design  Point 

The  execution  times  for  30  runs  of  each  of  the  17  design  points  which 
make  up  the  NOLH  design  of  experiments  utilized  by  this  study  were  recorded 
and  assessed.  The  time  to  run  one  design  point  ranged  from  2.03  to  10.52  hours, 
with  an  average  run  time  of  approximately  6.78  hours.  Total  run  time  for  the 
entire  experiment  was  approximately  1 15.34  hours.  These  and  other  descriptive 
statistics,  as  well  as  a  histogram  and  box  plot,  of  the  run  time  a  single  design 
point  are  summarized  in  Figure  32. 
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Figure  31 .  Histogram,  Box  Plot,  and  Descriptive  Statistics  of  the  Run 
Time  of  30  Iterations  of  a  Single  Design  Point 


Total  Run  Time 


Summary  Statistics 

Mean 
Std  Dev 
Std  Err  Mean 
Upper  95%  Mean 
Lower  95%  Mean 
N 

Minimum 

Maximum 

Median 


6.7847059 

3.0153402 

0.7313274 

8.3350508 

5.234361 

17 

2.03 

10.52 

6.47 


Horizontal  axis  shows  total  run  time,  units  are  in  hours. 
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V.  CONCLUSIONS 


A.  ANSWERS  TO  RESEARCH  QUESTIONS 

1.  Are  the  Results  Generated  by  NTFSM  Repeatable? 

NTFSM  is  a  stochastic  simulation  that  utilizes  seeded  random  number 
generation  to  produce  results  that  vary  from  run  to  run.  Theoretically,  identical 
scenarios  with  identical  seeds  should  produce  the  exact  same  results.  NTFSM’s 
ability  to  produce  identical  results  however  had  not  been  verified.  This  study 
conducted  two  experiments  that  tested  NTFSM’s  ability  to  produce  repeatable 
results.  The  first  of  which  tested  whether  or  not  identical  NTFSM  scenarios  which 
were  run  for  a  single  iteration  produced  identical  results.  The  second  experiment 
tested  whether  or  not  identical  NTFSM  scenarios  which  were  run  over  multiple 
iterations  produced  identical  results.  In  both  cases  it  was  found  that  identical 
NTFSM  scenarios  which  utilize  the  same  seed  produce  identical  results.  As  a 
consequence  of  the  experiments,  it  was  also  found  that  identical  NTFSM 
scenarios  that  do  not  utilize  the  same  seed  do  show  at  least  some  stochastic 
variation  in  their  results. 

2.  What  Is  the  General  Behavior  of  NTFSM’s  Main  Outputs? 

NTFSM  is  capable  of  producing  numerous  outputs  including  monthly  and 
yearly  loss,  gain  and  financial  cost  estimates  broken  down  by  rating,  paygrade, 
years  of  service,  and  gender.  The  scope  of  this  thesis  concentrated  on  the  loss 
and  gain  estimates  that  were  listed  in  the  Uncertainty-Years  report.  It  was  found 
that  when  fiscal  year  2014  data  is  used  to  project  one  year  into  the  future 
(FY2015)  the  EAOS  Losses,  Retirement  Losses,  Attrition  Losses,  Prior  Service 
Gains,  Recruit  Gains,  and  Recruit  Losses  outputs  are  approximately  normally 
distributed.  The  End  Strength  output  does  not  seem  to  match  any  of  the  common 
statistical  distributions  and  it  does  not  seem  to  contain  very  much  stochastic 
variability.  Additional  study  as  to  why  there  is  such  little  variability  in  End  Strength 
is  required. 
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3.  How  Sensitive  Are  NTFSM’s  Main  Outputs  to  Changes  in  its 
User-defined  Economic  Factors? 

Due  to  the  limited  computing  resources  available  on  the  NMPBS  testing 
server  on  which  NTFSM  is  currently  housed,  only  seven  of  NTFSM’s  12 
economic  coefficients  could  be  explored.  Any  effects  that  the  remaining  five 
economic  factors  have  on  NTFSM  output  were  not  captured  by  the  meta-models 
developed  in  this  thesis.  Also  due  to  computing  resources,  the  only  scenarios 
that  could  be  explored  by  this  thesis  were  those  which  utilized  fiscal  year  2014 
data  to  project  one  year  into  the  future  (FY2015).  The  meta-models  and 
sensitivities  reported  in  this  thesis  are  only  valid  for  scenarios  that  also  use  fiscal 
year  2014  data  to  project  one  year  into  the  future  (FY2015).  The  details  of  which, 
and  to  what  extent,  the  economic  coefficients  have  an  effect  on  the  NTFSM 
outputs  explored  are  contained  in  Chapter  V  and  Appendix  D  of  this  thesis.  Table 
5  shows  a  summary  of  which  of  the  NTFSM  economic  coefficients  explored  have 
an  effect  on  the  NTFSM  outputs  explored. 


Table  5.  Summary  of  NTFSM  Economic  Coefficients  that  Have 
an  Effect  on  the  NTFSM  Outputs  Explored 


Reenlistment 

Recruit  Losses 

Attrition  Losses 

Long  Extensions 

Unemployment 

Pay 

Unemployment 

Pay 

Unemployment 

Pay 

Unemployment 

Attrition  Losses 

NO 

NO 

NO 

NO 

YES 

YES 

NO 

Retirement  Losses 

NO 

NO 

NO 

NO 

YES 

NO 

NO 

Recruit  Losses 

YES 

NO 

NO 

NO 

YES 

NO 

YES 

EAOS  Losses 

YES 

NO 

NO 

NO 

NO 

NO 

YES 

Prior  Service  Gains 

NO 

NO 

NO 

YES 

YES 

YES 

YES 

Recruit  Gains 

YES 

NO 

NO 

NO 

YES 

NO 

NO 

End  Strength 

NO 

YES 

NO 

YES 

YES 

NO 

YES 

All  NTFSM  outputs  explored  seem  to  have  at  least  some  level  of 
sensitivity  to  the  seven  economic  factors  explored  with  the  exception  of  the  End 
Strength  output.  The  coefficients  of  the  parameters  of  the  meta-model  developed 
for  the  End  Strength  output  are  very  small,  which  means  the  parameters  of  the 
model  have  a  very  small  effect  on  NTFSM’s  End  Strength  output.  An  End 
Strength  model  that  uses  only  the  intercept  value  (265,278)  as  the  End  Strength 
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prediction  value,  although  much  simpler,  may  perform  just  as  well  as  the  meta¬ 
model  developed  for  all  intents  and  purposes. 

B.  RECOMMENDATIONS  FOR  FUTURE  STUDY 

This  study  acts  as  a  proof  of  concept  that  simulation  analysis  and  meta¬ 
modeling  techniques  can  be  applied  to  NTFSM  to  gain  useful  insight  on  the 
behavior  of  the  NTFSM  simulation.  This  thesis  is  constrained  in  the  scope  to 
which  these  simulation  analysis  and  meta-modeling  techniques  can  be  applied 
due  to  the  computing  resources  available  on  the  NMPBS  testing  server  on  which 
NTFSM  in  currently  housed.  The  SEED  Center,  however,  is  working  on 
transferring  NTFSM  to  their  computing  cluster.  This  will  greatly  increase  the 
amount  of  computing  resources  available  for  future  analysis  of  NTFSM.  To  build 
on  the  experiments  conducted  in  this  thesis,  it  is  recommended  that  all  12  of 
NTFSM’s  economic  coefficients  and  all  economic  conditions  be  explored  to 
better  understand  the  interactions  and  effects  that  could  not  be  captured  by  this 
thesis.  Also,  due  to  computing  resource  constraints,  the  results  of  this  thesis  only 
apply  to  NTFSM  scenarios  that  utilize  fiscal  year  2014  data  to  project  one  year 
into  the  future  (FY2015).  If  the  computing  resource  constraint  was  no  longer 
present,  however,  then  NTFSM  scenarios  that  utilize  the  full  spectrum  of  fiscal 
year  data  contained  in  NTFSM’s  data  repository  could  potentially  be  explored 
and  more  generalizable  results  could  be  produced. 
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APPENDIX  A.  LIST  OF  NTFSM  CAPABILITIES  MANDATED  BY 
THE  NAVY  TOTAL  FORCE  STRENGTH  MODEL  PROGRAM  PLAN 


The  following  information  was  taken  directly  from  the  Navy  Total  Force 
Strength  Model  Program  Plan  (Department  of  the  Navy,  2011) 


A  model  capability  is  defined  as  critical  and  fundamental  model 
functionality  desired  by  stakeholders.  The  Navy  Total  Force  Strength  Model 
capabilities  to  be  developed  include: 

•  Impact  of  LOS  into  forecast  of  future  inventory  by  paygrade. 
(CAP1 ) 

•  Incorporate  econometric  effects  of  losses  by  LOS  and  paygrade 
using  parameters  generated  by  the  Navy  Econometric  Modeling 
System  (NEMS)  to  the  greatest  extent  possible.  (CAP2) 

•  Enable  the  modeling  of  a  wide  variety  of  changes  to  policy  and 
estimate  their  impact.  (CAP3) 

•  Provide  ability  to  build  multiple  scenarios  either  for  a  specific  date 
of  for  the  Future  Years  Defense  Plan  (FYDP)  range,  including  the 
ability  for  users  to  modify  inputs  related  to  economics,  losses, 
gains,  advancements,  and  Navy  policy  with  limited  user  interaction. 
(CAP4) 

•  Provide  the  ability  to  perform  side-by-side  analysis  of  multiple 
scenarios  to  include  the  ability  to  visualize  and  compare  input 
variables.  (CAP5) 

•  Provide  the  ability  to  calculate  cost  and  associated  metrics  to 
include  total  cost,  promotion  and  accession  costs,  and  both 
aggregate  and  pay-grade  work-year  averages  of  a  set  of  strength 
plans,  including  validation/updates  to  underlying  cost  data.  (CAP6) 

•  Quantify  and  display  risk/uncertainty  in  forecasts  for  specific 
metrics  including  strength  and  costs.  Estimate  the  primary  sources 
of  risk/uncertainty  and  the  sensitivity  of  the  output  to  changes  in  the 
inputs.  (CAP7) 

•  Automated  comparison  of  strength  plans  versus  actual  execution, 
as  well  as  previous  plans  versus  current  plans,  including  the  ability 
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for  users  to  backcast  to  evaluate  the  impact  of  alternate  settings  on 
simulated  forecast  accuracy.  (CAP8) 

•  Provide  comparison  between  outputs  of  this  model  and  community- 
level  models.  (CAP9) 

•  Model  architecture  will  support  hosting  of  model,  scenarios,  (inputs, 
user  comments,  etc.)  and  outputs  in  a  secure  Navy  environment, 
such  as  the  Navy  Manpower,  Programming,  and  Budget  System 
(NMPBS),  and  will  support  data  and  reporting  requirements  from 
the  Office  of  the  Secretary  of  Defense  (OSD),  Director,  MPN 
Financial  Management  Division  (N10),  and  other  stakeholders 
minimizing  the  need  for  additional  transformations  or  rework. 
(CAP10) 

•  Existing  capabilities  of  the  existing  strength  model:  (CAP1 1 ) 

•  Generate  strength  plans  by  paygrade  and  month  for  a  range 
of  fiscal  years. 

•  Generate  scenarios  by  paygrade  and  month  at  varying 
points  of  the  execution  year  (i.e. .Actual  (A1 ,  A2,  A3,  etc). 

•  Forecast  total  Expiration  of  Active  Obligated  Service  (EAOS) 
actions  by  month  and  paygrade. 

•  Forecast  monthly  retirement  losses  by  paygrade  and  LOS. 

•  Forecast  attrition  losses  by  month  and  paygrade. 

•  Forecast  non-accession  gains  by  month  and  paygrade. 

•  Compute  total  recruit  gains  to  meet  fixed  end  strength. 

•  Compute  end  strength  given  fixed  accession  plan. 

•  Forecast  automatic  grade  movements. 

•  Forecast  advancement  plan  based  on  calculated  vacancies. 

•  Enhanced  capabilities  of  the  existing  strength  model  that  are 
approved  for  implementation.  (CAP12) 
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APPENDIX  B.  MONTHLY  SUMMARY  REPORTS  FOR 
REPEATABILITY  OF  NTFSM  SCENARIO  OUTPUT  EXPERIMENTS 


Monthly  Summary  reports  for  single  run  output  repeatability  assessment 
for  Experiment  One. 

The  scenarios  were  set  up  to  run  one  iteration  of  the  scenario  and  use 
fiscal  year  2014  Manpower  and  Personnel  data  to  project  one  year  into  the  future 
(FY2015).  All  scenario  inputs  were  kept  at  their  default  values  and  each  scenario 
used  the  same  5  digit  random  seed  (41701 ). 

Monthly  Summary  Report  for  the  First  Scenario  (seed  41701 ) 


Start:  FY15- OCT  Number  Of  FYs:  1  Population  Group:  Active  9  Executions:  1 
For  sims  that  do  not  start  in  Oct.  Report  will  use  history  for  missing  months  data 


Strength  Prev  Sep 

Oct 

Nov 

Dec 

Jan 

Feti 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total  with  OCs  266.703 

266,576 

266.463 

266.353 

266.226 

266.105 

265.992 

265.876 

265.764 

265,632 

265.526 

265.397 

265.277 

Enlisted  OCs  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Total  without  OCs  266.703 

266,578 

266.463 

266.353 

266.228 

266,105 

285.092 

265.876 

265.764 

265.832 

265.526 

265.397 

265.277 

WYA  by  Month  0 

266,640 

266.520 

266.408 

266.200 

266.166 

266.049 

265.934 

265.820 

265.808 

265.570 

265.462 

265,337 

Gains 

Oct 

NOV 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

PSG  Gains 

54 

30 

41 

61 

70 

42 

34 

34 

21 

31 

27 

24 

469 

ADOS 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Prior  Service 

27 

14 

11 

37 

38 

19 

12 

11 

7 

11 

5 

3 

195 

Other  Gains 

27 

16 

30 

24 

32 

23 

22 

23 

14 

20 

22 

21 

274 

NPS  Gains 

2.662 

2.288 

2.123 

2.733 

2.343 

2.065 

2.484 

2.058 

2.615 

3.471 

2.936 

2.526 

31.204 

TEAOS  0  to  3  Years 

0 

0 

5 

0 

0 

0 

1 

0 

0 

1 

1 

0 

8 

TEAOS  4  Years 

2.637 

2.255 

2.118 

2,720 

2.337 

2.040 

2.483 

2.927 

2.603 

3.448 

2.918 

2.508 

30,981 

TEAOS  5  Years 

17 

22 

0 

11 

1 

9 

13 

17 

11 

19 

14 

14 

148 

TEAOS  6+  Years 

8 

11 

0 

2 

5 

7 

7 

14 

1 

3 

5 

4 

67 

Total:  Gains 

2.716 

2.318 

2.164 

2.794 

2.413 

2.107 

2.518 

2.992 

2.636 

3.502 

2.963 

2.550 

31,673 

Losees 

Oct 

NOV 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

Attrition  Losses 

701 

680 

710 

880 

731 

676 

899 

1.147 

820 

1.040 

914 

827 

10.151 

Physical 

253 

280 

323 

437 

273 

238 

318 

307 

378 

534 

394 

336 

4.069 

Hardship 

17 

32 

12 

0 

24 

23 

20 

32 

12 

14 

9 

13 

217 

Cause 

408 

296 

278 

351 

327 

329 

441 

419 

302 

394 

348 

274 

4.167 

Miscelaneoos 

113 

81 

106 

02 

107 

86 

122 

389 

137 

98 

163 

204 

1.698 

EAOS  Losses 

1.268 

067 

880 

1.335 

1.197 

071 

1.150 

1.240 

1.205 

1.757 

1.274 

1.114 

14.387 

Zone  A  :  LOS  <=  6  years 

022 

664 

572 

979 

811 

617 

791 

837 

876 

1.339 

904 

768 

10.080 

Zone  B  :  LOS  7  to  10  years 

242 

213 

218 

258 

291 

250 

291 

307 

253 

307 

293 

264 

3.187 

Zone  C  :  LOS  1 1  to  14  years 

84 

74 

76 

86 

72 

91 

80 

79 

55 

86 

63 

62 

888 

Zone  D  :  LOS  15  to  20  years 

20 

15 

14 

10 

22 

13 

17 

17 

20 

23 

14 

18 

203 

ZoneE  :  LOS  >=21  years 

0 

1 

0 

2 

1 

0 

0 

0 

1 

2 

0 

2 

9 

Retirement  Losses 

411 

455 

402 

388 

368 

363 

343 

402 

473 

505 

543 

424 

5.077 

T  rainee  Losses 

373 

320 

273 

300 

238 

210 

233 

315 

261 

306 

361 

305 

3.504 

Total:  Losses 

2.843 

2.431 

2.274 

2.921 

2.534 

2.220 

2.634 

3.104 

2.768 

3.608 

3.092 

2,670 

33.099 
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Monthly  Summary  Report  for  the  Second  Scenario  (seed  41701 ) 

Start:  FY15-OCT  Number  Of  FYs:  1  Population  Group:  Active  #  Executions:  1 


For  sirrs  that  do  not  start  in  Oct.  Report  will  use  history  for  missing  months  data 


Strength  Prev  Sep 

Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total  with  OCs  266.703 

266.576 

266.463 

266.353 

266.226 

266.105 

265.992 

265.876 

265.764 

265.632 

265.526 

265.397 

265.277 

Enlisted  OCs  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Total  without  OCs  266.703 

266.570 

200.463 

206.353 

266.220 

260.105 

205.992 

205.870 

265.704 

265,032 

205.526 

205.307 

265.277 

WYA  by  Month  0 

260.040 

200.520 

200.408 

260.290 

200.100 

260.049 

205.934 

265.820 

265,098 

205.579 

205.402 

265.337 

Gains 

Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

PSG  Gains 

54 

30 

41 

01 

70 

42 

34 

34 

21 

31 

27 

24 

469 

ADOS 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Prior  Service 

27 

14 

11 

37 

38 

19 

12 

11 

7 

11 

5 

3 

195 

Other  Gains 

27 

10 

30 

24 

32 

23 

22 

23 

14 

20 

22 

21 

274 

NPS  Gains 

2.602 

2.288 

2.123 

2.733 

2.343 

2.005 

2.484 

2.958 

2.615 

3.471 

2.930 

2.520 

31.204 

TEAOS  0  to  3  Years 

0 

0 

5 

0 

0 

0 

1 

0 

0 

1 

1 

0 

8 

TEAOS  4  Years 

2.637 

2.255 

2.118 

2,720 

2.337 

2.049 

2.403 

2.927 

2.603 

3.448 

2.916 

2.508 

30.981 

TEAOS  5  Years 

17 

22 

0 

11 

1 

9 

13 

17 

11 

16 

14 

14 

148 

TEAOS  6+  Years 

8 

11 

0 

2 

5 

7 

7 

14 

1 

3 

5 

4 

67 

Total:  Gains 

2.716 

2.318 

2.164 

2.794 

2.413 

2.107 

2.518 

2.992 

2.636 

3.502 

2.963 

2.550 

31.673 

Losses 

Oct 

NOV 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

Attrition  Losses 

701 

089 

719 

889 

731 

076 

899 

1.147 

826 

1.040 

914 

827 

10.151 

Physical 

253 

280 

323 

437 

273 

238 

310 

307 

378 

534 

394 

336 

4,069 

Hardship 

17 

32 

12 

0 

24 

23 

20 

32 

12 

14 

9 

13 

217 

Cause 

408 

290 

278 

351 

327 

329 

441 

419 

302 

394 

348 

274 

4.107 

Mtscelaneous 

113 

81 

100 

92 

107 

80 

122 

389 

137 

98 

103 

204 

1,698 

EAOS  Losses 

1.268 

907 

880 

1.335 

1.107 

071 

1.159 

1.240 

1.205 

1.757 

1.274 

1.114 

14,367 

Zone  A  :  LOS  <=  0  years 

022 

004 

572 

079 

811 

017 

791 

837 

876 

1,339 

904 

768 

10,080 

Zone  B  :  LOS  7  to  10  years 

242 

213 

218 

258 

291 

250 

291 

307 

253 

307 

293 

204 

3,187 

Zone  C  :  LOS  1 1  to  14  years 

84 

74 

70 

80 

72 

91 

00 

79 

55 

80 

03 

62 

888 

Zone  D  :  LOS  15  to  20  years 

20 

15 

14 

10 

22 

13 

17 

17 

20 

23 

14 

18 

203 

Zone  E  :  LOS  >=  21  years 

0 

1 

0 

2 

1 

0 

0 

0 

1 

2 

0 

2 

9 

Retirement  Losses 

411 

455 

402 

388 

368 

303 

343 

402 

473 

505 

543 

424 

5.077 

T  rainee  Losses 

373 

320 

273 

309 

238 

210 

233 

315 

201 

300 

301 

305 

3.504 

Total:  Losses 

2.843 

2.431 

2.274 

2.921 

2.534 

2.220 

2.634 

3.104 

2.768 

3.608 

3.092 

2,670 

33.099 
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Monthly  Summary  reports  for  single  run  output  repeatability  assessment 
for  Experiment  Two. 

The  scenarios  were  set  up  to  run  one  iteration  of  the  scenario  and  use 
fiscal  year  2014  Manpower  and  Personnel  data  to  project  one  year  into  the  future 
(FY2015).  All  scenario  inputs  were  kept  at  their  default  values  and  each  scenario 
used  the  same  5  digit  random  seed  (66389). 

Monthly  Summary  Report  for  the  First  Scenario  (seed  66389) 

Start:  FY1 5 -OCT  Number  Of  FYs:  1  Population  Group:  Active  #  Executions:  1 


Strength  Prev  Sep 

Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

S«p 

Total  with  OCs  266.703 

266.591 

266.465 

266.341 

266.229 

266.106 

265.990 

265.869 

265.757 

265,632 

265.516 

265.395 

265.282 

Enlisted  OCs  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Total  without  OCs  266,703 

266.501 

260,465 

266.341 

266.229 

266.106 

265.990 

265.869 

265.757 

265.632 

265.516 

265.385 

265.282 

WYA  by  Month  0 

266.047 

266.528 

266.403 

266.285 

266.108 

266.048 

265.830 

265.813 

265.095 

265.574 

265.456 

265,339 

Gains 

Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

PSG  Gains 

04 

57 

35 

54 

58 

38 

48 

23 

31 

32 

23 

25 

488 

ADOS 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Prior  Service 

41 

32 

13 

27 

28 

12 

17 

0 

11 

3 

3 

3 

197 

Other  Gains 

23 

25 

22 

27 

28 

26 

31 

17 

20 

29 

20 

22 

291 

NPS  Gains 

2.633 

2.200 

2.086 

2.681 

2.426 

2.115 

2.373 

2,969 

2.473 

3.567 

2.920 

2.579 

31.112 

TEAOS  0  to  3  Years 

0 

0 

5 

0 

0 

0 

1 

0 

0 

0 

1 

0 

7 

TEAOS  4  Years 

2.617 

2.263 

2.081 

2,668 

2.418 

2.109 

2.349 

2.944 

2.461 

3.553 

2,908 

2.570 

30.947 

TEAOS  5  Years 

6 

18 

0 

10 

3 

0 

6 

13 

10 

14 

8 

3 

101 

TEAOS  6+  Years 

7 

9 

0 

3 

5 

0 

17 

12 

2 

0 

2 

0 

57 

TotaJ:  Gains 

2.697 

2.347 

2.121 

2.735 

2.484 

2.153 

2.421 

2,992 

2.504 

3.599 

2,943 

2.604 

31.600 

L08Se8 

Oct 

NOV 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

Attrition  Losses 

803 

681 

719 

936 

744 

721 

850 

1,173 

830 

1.058 

800 

863 

10.194 

Physical 

248 

277 

306 

458 

267 

251 

307 

319 

391 

508 

391 

379 

4.103 

Hardship 

10 

28 

16 

5 

25 

12 

18 

23 

14 

23 

13 

11 

204 

Cause 

422 

275 

307 

391 

322 

346 

423 

443 

291 

416 

261 

271 

4.168 

Mtsoelaneous 

110 

111 

90 

82 

130 

112 

108 

388 

134 

111 

135 

202 

1.719 

EAOS  Losses 

1.203 

1.021 

860 

1.263 

1.216 

988 

1.141 

1.217 

1.123 

1.788 

1.374 

1,086 

14.341 

Zone  A :  LOS  <=  0  years 

027 

683 

570 

900 

820 

070 

778 

797 

784 

1.326 

984 

748 

9.985 

Zone  B  :  LOS  7  to  10  years 

238 

248 

205 

247 

307 

227 

284 

315 

271 

335 

318 

260 

3.256 

Zone  C  :  LOS  1 1  to  14  years 

78 

70 

70 

103 

66 

80 

64 

82 

56 

96 

55 

63 

883 

Zone  D  :  LOS  15  to  20  years 

20 

18 

15 

11 

20 

11 

10 

22 

11 

29 

17 

14 

205 

Zone  E  :  LOS  >=  21  years 

0 

0 

0 

2 

3 

0 

1 

1 

1 

3 

0 

1 

12 

Retirement  Losses 

411 

454 

402 

387 

365 

361 

322 

420 

451 

543 

533 

456 

5.105 

T  rainee  Losses 

332 

307 

204 

261 

282 

189 

223 

294 

225 

325 

357 

312 

3.381 

Total:  Losses 

2.809 

2.473 

2.245 

2.847 

2.607 

2.269 

2.542 

3.104 

2.629 

3.715 

3.064 

2,717 

33.021 
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Monthly  Summary  Report  for  the  Second  Scenario  (seed  66389) 

Start:  FY15-OCT  Number  Of  FYs:  1  Population  Group:  Active  9  Executions:  1 


For  sims  that  do  not  start  in  Oct,  Report  will  use  history  for  missing  months  data 


Strength  Prev  Sep 

Oct 

NOV 

Dec 

Jan 

Feb 

P4ar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total  with  OCs  266.703 

266,591 

266.465 

266.341 

266.229 

266,106 

265.990 

265.869 

265.757 

265.632 

265.516 

265.395 

265.282 

Enlisted  OCs  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Total  without  OCs  266,703 

266,591 

266.465 

266.341 

266,229 

260,106 

205,990 

265.869 

265,757 

265,032 

205.516 

265.395 

265,282 

WYA  by  Month  0 

266,647 

260.528 

206.403 

266,285 

260.108 

200,048 

205.930 

265,813 

205,095 

205,574 

205.450 

265,339 

Gains 

Oct 

Nov 

Dec 

Jan 

Fet> 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

PSG  Gains 

64 

57 

35 

54 

58 

38 

48 

23 

31 

32 

23 

25 

488 

ADOS 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Prior  Service 

41 

32 

13 

27 

29 

12 

17 

0 

11 

3 

3 

3 

197 

Other  Gains 

23 

25 

22 

27 

29 

20 

31 

17 

20 

29 

20 

22 

291 

NPS  Gains 

2.633 

2.290 

2,080 

2,681 

2.426 

2.115 

2,373 

2,969 

2.473 

3.507 

2,920 

2,579 

31,112 

TEAOS  0  to  3  Years 

0 

0 

5 

0 

0 

0 

1 

0 

0 

0 

1 

0 

71 

TEAOS  4  Years 

2.617 

2.203 

2,061 

2,668 

2.418 

2.109 

2,349 

2,944 

2.401 

3,553 

2,908 

2,576 

30,947 

TEAOS  5  Years 

e 

18 

0 

10 

3 

0 

6 

13 

10 

14 

9 

3 

101 

TEAOS  6+  Years 

7 

9 

0 

3 

5 

0 

17 

12 

2 

0 

2 

0 

57 

Total:  Gains 

2.697 

2.347 

2.121 

2.735 

2.484 

2.153 

2,421 

2,992 

2.504 

3,599 

2.943 

2.664 

31.600 

Losees 

Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

Attrition  Losses 

803 

691 

719 

936 

744 

721 

850 

1.173 

830 

1.058 

800 

863 

10,194 

Physical 

249 

277 

306 

458 

267 

251 

307 

319 

391 

508 

391 

379 

4,103 

Hardship 

16 

28 

10 

5 

25 

12 

18 

23 

14 

23 

13 

11 

204 

Cause 

422 

275 

307 

391 

322 

340 

423 

443 

291 

410 

201 

271 

4,168 

Mtscelaneous 

116 

111 

90 

82 

130 

112 

108 

388 

134 

111 

135 

202 

1.719 

EAOS  Losses 

1.263 

1.021 

860 

1,263 

1.216 

988 

1.141 

1.217 

1.123 

1.789 

1,374 

1,086 

14.341 

Zone  A  :  LOS  <=  6  years 

927 

083 

570 

900 

820 

070 

770 

797 

784 

1.320 

984 

748 

9.985 

Zone  B  :  LOS  7  to  10  years 

238 

249 

205 

247 

307 

227 

284 

315 

271 

335 

318 

260 

3,256 

Zone  C  :  LOS  1 1  to  1 4  years  78 

70 

70 

103 

66 

80 

64 

82 

56 

90 

55 

63 

883 

Zone  D:  LOS  15  to  20  years  20 

19 

15 

11 

20 

11 

10 

22 

11 

29 

17 

14 

205 

Zone  E  :  LOS  >=  21  years 

0 

0 

0 

2 

3 

0 

1 

1 

1 

3 

0 

1 

12 

Retirement  Losses 

411 

454 

402 

387 

365 

301 

322 

420 

451 

543 

533 

456 

5,105 

T  rainee  Losses 

332 

307 

264 

261 

282 

199 

223 

294 

225 

325 

357 

312 

3.381 

Total:  Losses 

2.809 

2.473 

2.245 

2.847 

2.607 

2.269 

2.542 

3.104 

2.629 

3.715 

3,064 

2.717 

33.021 

62 


Monthly  Summary  reports  for  multiple  run  output  repeatability  assessment. 

The  scenarios  were  set  up  to  simulate  five  iterations  of  the  scenario  and 
use  fiscal  year  2014  Manpower  and  Personnel  data  to  project  one  year  into  the 
future  (FY2015).  All  scenario  inputs  were  kept  at  their  default  values  and  each 
scenario  used  the  same  five  digit  random  seed  (37295). 

Monthly  Summary  Report  for  the  First  Scenario  (seed  37295) 

Start:  FY1 5  -OCT  Number  Of  FYs:  1  Population  Group:  Active  9  Executions:  5 


For  sims  that  do  not  start  in  Oct.  Report  will  use  history  for  missing  months  data 


Strength  Prev  Sep 

Oct 

NOV 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

S«p 

Total  with  OCs  266.703 

266.583 

266.463 

266.348 

266.228 

266.108 

265.990 

265.871 

265.748 

265,635 

265.517 

265.397 

265.278 

Enlisted  OCs  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

T otal  without  OCs  206.703 

206.583 

200.463 

208.348 

266.228 

200.108 

205.990 

205.871 

265.748 

205.035 

205.517 

205.397 

265.278 

WYA  by  Month  0 

206.643 

200.523 

200.406 

260.288 

200,168 

200.049 

205.931 

265.810 

205.092 

205.570 

205.457 

265.337 

Game 

Oct 

NOV 

Ok 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

PSG  Gains 

02 

39 

39 

87 

55 

48 

47 

37 

21 

35 

20 

24 

500 

ADOS 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0| 

Prior  Service 

34 

24 

11 

30 

25 

17 

17 

11 

5 

4 

3 

5 

187 

Other  Gains 

28 

14 

28 

37 

29 

31 

30 

20 

10 

31 

23 

19 

312 

NPS  Gains 

2.010 

2.322 

2.108 

2.072 

2.417 

2.071 

2.435 

2.877 

2.502 

3.437 

2.910 

2.528 

30.963 

TEAOS  0  to  3  Years 

0 

0 

2 

0 

0 

0 

1 

0 

0 

1 

2 

0 

6 

TEAOS  4  Years 

2.590 

2.292 

2.100 

2.061 

2.402 

2,064 

2.414 

2.842 

2.542 

3.419 

2.900 

2.512 

30.757 

TEAOS  5  Years 

13 

23 

0 

9 

7 

5 

11 

22 

18 

13 

7 

11 

138 

TEAOS  6+  Years 

8 

7 

0 

2 

8 

2 

9 

13 

2 

4 

1 

4 

02 

Total:  Gains 

2.678 

2.361 

2.147 

2.739 

2.472 

2.120 

2.482 

2.914 

2.583 

3.472 

2.942 

2.551 

31.462 

Loews 

Oct 

Nov 

Ok 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total 

Attrition  Losses 

772 

083 

711 

908 

738 

710 

869 

1.148 

831 

1.032 

829 

825 

10.055 

Physical 

259 

248 

290 

441 

294 

243 

300 

287 

371 

507 

379 

348 

3.979 

Hardship 

19 

25 

12 

7 

15 

19 

24 

28 

11 

13 

14 

12 

200 

Cause 

374 

310 

309 

357 

309 

345 

420 

430 

302 

420 

298 

283 

4.162 

Mscelaneous 

121 

99 

93 

103 

119 

103 

119 

390 

147 

92 

139 

182 

1.714 

EAOS  Losses 

1.269 

1.019 

878 

1.282 

1233 

959 

1.102 

1.206 

1.147 

1.734 

1.340 

1.070 

14.305 

Zone  A  LOS  <=  0  years 

920 

684 

584 

939 

820 

645 

790 

799 

810 

1.307 

929 

720 

9,959 

Zone  B  :  LOS  7  to  10  years 

249 

240 

210 

238 

323 

225 

282 

304 

240 

317 

324 

205 

3.235 

Zone  C  :  LOS  1 1  to  14  years  81 

70 

03 

88 

72 

74 

02 

86 

70 

87 

77 

63 

893 

Zone  D  :  LOS  15  to  20  years  20 

19 

14 

17 

15 

14 

22 

17 

15 

22 

10 

20 

211 

Zone  E  :  LOS  >=  21  years 

0 

0 

0 

1 

3 

1 

0 

0 

0 

1 

0 

1 

8 

Retirement  Losses 

411 

455 

401 

387 

365 

363 

342 

399 

408 

502 

524 

471 

5.087 

T rainee  Losses 

340 

325 

272 

283 

255 

205 

228 

285 

250 

322 

303 

304 

3.440 

Total:  Losses 

2.798 

2.481 

2.262 

2.860 

2.592 

2.237 

2.601 

3.037 

2.696 

3.591 

3,062 

2.670 

32.887 

63 


Monthly  Summary  Report  for  the  Second  Scenario  (seed  37295) 

Start:  FY15-OCT  Number  Of  FYs:  1  Population  Group:  Active  #  Executions:  5 


For  sirns  that  do  not  start  n  Oct,  Report  will  use  history  for  missing  months  data 


Strength  Prev  Sep 

Oct 

NOV 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Total  with  OCs  266.703 

266.583 

266,463 

266.348 

266.228 

266.108 

265.990 

265.871 

265.748 

265,635 

265.517 

265.397 

265,278 

Enlisted  OCs  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Total  without  OCs  266,703 

266,583 

266,463 

266.348 

266,228 

266,108 

285.990 

285.871 

265,748 

265,635 

265.517 

265.307 

265.278 

WYA  by  Month  0 

266,643 

266,523 

266.406 

266,288 

266,168 

286.049 

265.931 

265,810 

265,892 

285,576 

285.457 

265,337 

Gains 

Oct 

Nov 

Dec 

Jan 

Fob 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sop 

Total 

PSG  Gains 

62 

39 

39 

67 

55 

48 

47 

37 

21 

35 

26 

24 

500 

ADOS 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Prior  Service 

34 

24 

11 

30 

25 

17 

17 

11 

5 

4 

3 

5 

187 

Other  Gains 

28 

14 

28 

37 

29 

31 

30 

26 

16 

31 

23 

19 

312 

NPS  Gains 

2.616 

2.322 

2,108 

2,672 

2.417 

2.071 

2,435 

2.877 

2.562 

3,437 

2,916 

2,528 

30,963 

TEAOS  0  to  3  Years 

0 

0 

2 

0 

0 

0 

1 

0 

0 

1 

2 

0 

6 

TEAOS  4  Years 

2.506 

2292 

2,106 

2,661 

2.402 

2.064 

2.414 

2,842 

2.542 

3,419 

2,906 

2,512 

30.757 

TEAOS  5  Years 

13 

23 

0 

9 

7 

5 

11 

22 

18 

13 

7 

11 

138 

TEAOS  6+  Years 

8 

7 

0 

2 

8 

2 

9 

13 

2 

4 

1 

4 

62 

Total:  Gains 

2.678 

2.361 

2.147 

2.739 

2.472 

2.120 

2.482 

2.914 

2.583 

3.472 

2,942 

2.551 

31.462 

Losses 

Oct 

Nov 

Doc 

Jan 

Fob 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sop 

Total 

Attrition  Losses 

772 

683 

711 

908 

738 

710 

869 

1.148 

831 

1.032 

829 

825 

10,055 

Physical 

259 

248 

296 

441 

294 

243 

306 

287 

371 

507 

379 

348 

3,979 

Hardship 

19 

25 

12 

7 

15 

19 

24 

28 

11 

13 

14 

12 

200 

Cause 

374 

310 

309 

357 

309 

345 

420 

436 

302 

420 

298 

283 

4,162 

Miscelaneous 

121 

99 

93 

103 

110 

103 

119 

396 

147 

92 

139 

182 

1,714 

EAOS  Losses 

1269 

1.019 

878 

1.282 

1233 

959 

1,182 

1,206 

1.147 

1.734 

1,346 

1,070 

14,305 

Zone  A  :  LOS  <=  6  years 

920 

684 

584 

939 

820 

645 

796 

799 

816 

1,307 

929 

720 

9.959 

Zone  B  :  LOS  7  to  10  years 

249 

246 

218 

238 

323 

225 

282 

304 

246 

317 

324 

265 

3,235 

Zone  C  :  LOS  1 1  to  1 4  years 

81 

70 

83 

88 

72 

74 

82 

86 

70 

87 

77 

63 

893 

Zone  D  :  LOS  15  to  20  years 

20 

19 

14 

17 

15 

14 

22 

17 

15 

22 

16 

20 

211 

Zone  E  LOS  >=  21  years 

0 

0 

0 

1 

3 

1 

0 

0 

0 

1 

0 

1 

8 

Retirement  Losses 

411 

455 

401 

387 

365 

363 

342 

399 

468 

502 

524 

471 

5.087 

T  rainee  Losses 

346 

325 

272 

283 

255 

205 

228 

285 

250 

322 

363 

304 

3  440 

Total:  Losses 

2.798 

2.481 

2.262 

2.860 

2.592 

2.237 

2.601 

3.037 

2.696 

3.591 

3,062 

2.670 

32.887 

64 


APPENDIX  C.  A  SUMMARY  OF  THE  DISTRIBUTION  OF 
RETIREMENT  LOSSES,  RECRUIT  LOSSES,  EAOS  LOSSES, 
PRIOR  SERVICE  GAINS,  AND  RECRUIT  GAINS 


65 


EAOS  Losses 


2.5% 

0.5% 


Summary  Statistics 


14523 

14523 

14519 

14464 

14381.8 

14312.5 
14250 

14199.6 

14124.8 
14034 
14034 


Fitted  Normal 


14316.770 

97.697 

9.770 

14336.155 

14297.385 

100.000 


Parameter  Estimates 


14316.770 

97.697 


Lower  95%  Upper  95% 

14297.385  14336.155 

85.779  113.492 


!log(Likelihood)  =  1199.16188059038 


Goodness-of-Fit  Test 


Recruit  Gains 


■C^ZF 


Summary  Statistics 

31368 

Mean 

31007.250 

31368 

Std  Dev 

158.527 

31306.1 

Std  Err  Mean 

15.853 

31216 

Upper  95%  Mean 

31038.705 

31138,3 

Lower  95%  Mean 

30975,795 

31007 

30904.8 

30790.2 

30673.5 

30608 

30608 

N 

100.000 

Fitted  Normal 


Parameter  Estimates 

Type  Parameter  Estimate  Lower  95% 

Location  p  31007.250  30975.795 

Dispersion  o  158.527  139.188 

-2log(Likelihood)  =  1295.97311732681 


Upper  95% 

31038.705 

184.157 


Goodness-of-Fit  Test 

Shapiro-Wilk  W  Test 

W  Prob<W 

0,993  0.889 


Note:  Ho  =  The  data  is  from  the  Normal  dis 
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APPENDIX  D.  SUMMARIES  OF  REMANING  META-MODELS 


Retirement  Losses 


Summary  of  Fit 


RSquare 

0.9472 

RSquare  Adj 

0.9397 

Root  Mean  Square  Error 

11.162 

Mean  of  Response 

5077.4 

Observations  (or  Sum  Wgts) 

17 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

5097.715 

4.073 

1251.7 

<.000f 

Attrite  U 

2.117 

0.147 

14.37 

<.oooi; 

Attrite  U*Attrite  U 

-0.060 

0.009 

-6.69 

<.0001 5 

Prior  Service  Gains 


Summary  of  Fit 

RSquare 

RSquare  Adj 

Root  Mean  Square  Error 

Mean  of  Response 

Observations  (or  Sum  Wgts) 

0.8752 

0.7781 

1.374 

482.41 

17 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

484.96 

0.5189 

934.67 

<.0001 

Long  Ext  U 

-0.196 

0.0563 

-3.49 

0.0069  * 

Recruit  P 

-0.035 

0.0184 

-1.91 

0.0889 

Attrite  U 

0.0231 

0.021 

1.10 

0.2989 

Attrite  P 

-0.006 

0.0201 

-0.29 

0.7775 

Attrite  U*Attrite  P 

0.0049 

0.0011 

4.48 

0.0015  * 

Long  Ext  U*Long  Ext  U 

-0.008 

0.0012 

-6.40 

0.0001  * 

Long  Ext  U*Long  Ext  U*Long  Ext  U 

0.0003 

0.0001 

3.31 

0.0090* 
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Recruit  Gains 


Summary  of  Fit 


RSquare 

0.9828 

RSquare  Adj 

0.9788 

Root  Mean  Square  Error 

2808.2 

Mean  of  Response 

36889 

Observations  (or  Sum  Wgts) 

17 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

29591 

1025 

28.88 

<.0001 

Reenlist  U 

399.14 

37.07 

10.77 

<.0001* 

Attrite  U 

-857.3 

37.07 

-23.12 

<.0001* 

Attrite  U*Attrite  U 

21.625 

2.268 

9.53 

<.0001* 

Recruit  Losses 

Summary  of  Fit 

RSquare 

0.9883 

RSquare  Adj 

0.9844 

Root  Mean  Square  Error 

214.82 

Mean  of  Response 

3991.4 

Observations  (or  Sum  Wgts) 

17 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

3330 

78.38 

42.48 

<.0001 

Reenlist  U 

34.89 

2.836 

12.30 

<.0001* 

Long  Ext  U 

6.88 

2.836 

2.43 

0.0320* 

Attrite  U 

-76.7 

2.836 

-27.03 

<.0001* 

Attrite  U*Attrite  U 

1.961 

0.1735 

11.30 

<.0001* 
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APPENDIX  E.  COMPARISON  PLOTS  FOR  REMAINING  META¬ 
MODELS 


Comparison  Plots  for  Retirement  Losses 


Observation  Prediction 

Test  Scenario  1  (Upper  Bound  Test) 


Observation  Prediction 

Test  Scenario  2  (Lower  Bound  Test) 


Observation  Prediction 

Test  Scenario  3  (Random  Test) 
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Comparison  Plots  for  Attrition  Losses 


2000 


Prediction 


Observation 

Test  Scenario  1  (Upper  Bound  Test) 


Prediction 


Observation  r.un 

Test  Scenario  2  (Lower  Bound  Test) 


Comparison  Plots  for  Prior  Service  Gains 


£  480 


Observation  Prediction 

Test  Scenario  2  (Lower  Bound  Test) 


Observation  Prediction 

Test  Scenario  1  (Upper  Bound  Test) 


Observation  Prediction 

Test  Scenario  3  (Random  Test) 
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Comparison  Plots  for  Recruit  Gains 


Observation  Prediction 

Test  Scenario  1  (Upper  Bound  Test) 


480 


Observation  Prediction 

Test  Scenario  3  (Random  Test) 


Comparison  Plots  for  Recruit  Losses 


7000 

6500 


„  6000 
£ 

°  5500 

3 

S  5000 

O t 

4500  • _ 

4000  _  — * — 

^3  LJ 

3500  - -  - - 

Observation  Prediction 

Test  Scenario  1  (Upper  Bound  Test) 


71 


Comparison  Plots  for  EAOS  Losses 


23500 

23000 

22500 


S  22000 

o 

§  21500 

21000 
20500 
20000 


Observation  Prediction 

Test  Scenario  3  (Random  Test) 
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